Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
Programming Software United States IT Technology

Tracking the IT Job Market with a Bot 166

atlantageek writes "Is the IT job market improving? Is the growth in Unix or Windows? Should I study Data Warehousing or E-Commerce? Identify the recent trends with CJ Miner, a small tool I've written that has been monitoring the Computer Jobs website for the last year."
This discussion has been archived. No new comments can be posted.

Tracking the IT Job Market with a Bot

Comments Filter:
  • Don't you mean (Score:3, Insightful)

    by Anonymous Coward on Saturday July 09, 2005 @08:20PM (#13023473)
    "track how the computerjobs.com website has been doing"?
    • Re:Don't you mean (Score:5, Informative)

      by Scoria ( 264473 ) <`slashmail' `at' `initialized.org'> on Saturday July 09, 2005 @08:26PM (#13023496) Homepage
      "track how the computerjobs.com website has been doing"?

      That was also my interpretation of this project. I'm afraid that computerjobs.com wouldn't necessarily represent the entire IT market, but rather a very small percentage of it. The software would be limited to indicating various demographics at computerjobs.com, perhaps arguably and tentatively serving to indicate the "competency level" of their members. Without data from many sources, however, you couldn't hope to provide an accurate impression of the overall market.

      Maybe the programmer should sell the collected data back to them. ;-)
    • For at least a somewhat representative sample, one one need to add: Yahoo HotJobs, Monster, Dice and CareerBuilder. And maybe then we can start talking....
    • by mthreat ( 632318 )

      Or you can use indeed.com [indeed.com], which lets you search all jobs within the last 30 days from almost a thousand job sites (including computerjobs.com).

      You don't even have to visit the site to check for new jobs -- it has RSS feeds and email alerts for new jobs that match your search criteria.

      Or if you're really ambitious, use their free XML API [indeed.com] and do whatever you want with the data.

  • Quick... (Score:2, Funny)

    by ActionJesus ( 803475 )
    Ive written an advanced bot, plz hire me.
  • check out the business cycle!
  • More info needed (Score:3, Interesting)

    by randall_burns ( 108052 ) <randall_burns@@@hotmail...com> on Saturday July 09, 2005 @08:28PM (#13023510)
    1) How many of these ads are actually real?

    2) What do these jobs mean in terms of disposable income?

    • Newsgroups (where the posting is free) are certainly populated by bots posting the same jobs over and over, or even bots posting ads for their sites that list job harvested from other lists by bots... I'd imagine that sites like monster, etc, that charge companies for listing jobs would have a much higher ratio of real jobs. (There are companies that stay in a state of perpetual hiring for the same job for months and pay for the listings. Weird.)

      As for income, couldn't say, but I do get the feeling that th

    • by Seumas ( 6865 ) *
      Considering that it shows about 500,000 jobs in the Denver area right now (which would mean something like 25% of the population - man woman and child - are working in IT departments), I would say very few.

      Anyway, who the hell actually uses any sort of service or website to find a tech job above anything but entry level? That's what contacts and networking are for. You find yourself unemployed or looking for a new job and you put your feelers out to all your friends and colleagues who have moved to other c
    • #3) How many of these are actually unique jobs versus various recruiter spews?

      Seen many dupes on CJ.. Slashdot doesn't have the market cornered and CJ's a close second.
    • by reporter ( 666905 ) on Saturday July 09, 2005 @09:30PM (#13023760) Homepage
      The market for information-technology (IT) jobs does not operate according to the laws of economics. Allow me to explain. A shortage of labor is a normal market force, and government should not intervene to counteract this force. Two of the effects of a shortage is (1) to boost wages and (2) improve working conditions.

      However, whenever a shortage of labor occurs in the IT market, the government consistently intervenes by importing H-1B workers to fix this shortage. As a result, the growth in wages is damaged. Working conditions (like working 60+ hours per week) do not improve.

      Any perceived shortage in the market for IT labor is illusory. If this shortage were real, it would be short-lived, due to government intervention.

      By the way, we see the same phenomenon in the market for unskilled labor: e.g. picking vegetables and fruits. The government fixes this shortage by allowing illegal aliens to flood this market for unskilled labor. As a result, wages (hovering around $5.00 per hour for fruit-picking in Southern California) never rise. Working conditions (like standing for more than 9 hours per day in the strawberry fields) never improve.

      The rub is that politicians do not care about Washington's gross tampering in and bludgeoning of a (relatively) free market like the USA. Washington is eager to fix shortages of labor. However, Washington rarely fixes shortages of jobs by, for example, creating more government jobs. The interests of Washington are not aligned with the hopes and aspirations of middle America.

      We should close the American market to (relatively) non-free markets like India, China, and Mexico. Further, the American market should be flung wide open to (relatively) free markets like Eastern/Western Europe, Canada, and Japan. Free trade is good -- only when we are trading with other societies that maintain (relatively) free markets.

      • Its a little trickier than that. US citizenship has _real_ value in the world scence. That means that lots of folks are willing to do unpleasant work to get a shot at US citizenship/permanent residency. Right now, the ways to do that are:

        1) become an H-1b/L-1 IT worker

        2) join the US military

        3) become an a working illegal alien-and bide your time until the next amnesty.
        • Surely, you have to be a US citizen before you can join the military?
          • Foreign legion (Score:2, Informative)

            by tepples ( 727027 )

            Surely, you have to be a US citizen before you can join the military?

            Nope. Non-citizens may enlist in the U.S. armed services [todaysmilitary.com]. Think about it: France has a foreign legion [foreignlegionlife.com]; why can't the USA?

          • You don't have to be a citizen to enlist in the US armed forces, but you do in order to be an officer.
          • You can become a member of the US military without being a US citizen - they even go so far as to advertize it to aliens as a way to work towards citizenship! When I was an F1A foreign student (getting my Master's at SMSU, in Springfield MO), I received several recruiting brochures.

            What's worse, is that non-resident aliens are signed up for Selective Service and could in theory be drafted to fight for the US without any real ties to it other than doing a degree there! I was positively shocked to find out t

      • by Anonymous Coward
        You talk of Free market and closing the US job market to non-free countries like India, China, Mexico etc. I think your knowledge of free economics is pretty stunted. If you are talking about computers and IT industruy, India is a very very open market and one where the government doesnt interfere at all, unlike other industries and unlike the US.

        If you talk of general industry - then lets pick Steel - a very very regulated markket in the US with many stipulations to close free trade with other more compet
      • by Anonymous Coward
        Open markets like Europe... where the average cow earns US$2.50 a day in subsidies?

        Or Japan... where the average cow earns US$7.50 a day in subsidies?

        Link to article [indianexpress.com]
      • OK, Mr. Troll, I'll bite:

        1) There's nothing that says that a shortage in IT workers should result in an improvement in working conditions (defined in your post as a reduction in working hours). In fact, if there is a shortage of workers, the expectations for hours worked individually should increase, not decrease. Nobody's going to say "gee, Jenkins, we're so happy to have one of the few available experts on our staff that we want you to go home at 4:00 every day."

        2) Washington rarely fixes shortages
        • There's nothing that says that a shortage in IT workers should result in an improvement in working conditions (defined in your post as a reduction in working hours). In fact, if there is a shortage of workers, the expectations for hours worked individually should increase, not decrease.

          No, because if there really was a shortage, one could threaten to leave to work for a 40-hour company, and the biz would have to comply or start another difficult hunt.

          (BTW, there is no "IT shortage". Biz lobbyists made i
          • What you're saying may happen in individual cases, but overall the expectation for each worker would increase. If there's a shortage of labor, then by definition there's too much work for too few people, and in the short-term, increased hours is the most likely scenario...
      • I wouldn't worry too much. With the regulations put in place since September 11th, the U.S. is not as favorable a destination as it once was. As word spreads on how difficult it is for a non-citizen to cross the border even when all the papers are legal, fewer qualified workers will want to move to the U.S. Wages and working conditions will have to improve to fill the shortage.
      • Raising everyone's income will incite inflation, and you'll be no better off. In capitalism, someone has to get the shit end of the stick because it's built into the system. Communism didn't work. You'll have to think of a new system.
      • Jobs going to foreign countries or foreigners coming here to work ?

        And btw, "free market" is just an idealized abstraction [wikipedia.org] - with the risk of burning some karma I'll say it's pretty crude one, actually.

        The notion of "free market" was developed during the Enlightenment or "Age or Reason", when Europe started to attribute everything to science and reason (after the Middle Ages attributed everything to the will of God). As a consequence, it is based on the fact that players in the market make their decisi

        • Interesting point but things like H-1 visas and letting illegal immigrants work in the US when there's a labor shortage can hurt the corresponding industries in the US.

          Wages won't go up, because there's suddenly more workers available. Thus, it won't lure new workers in from other industries where there may be a more abundance of workforce right here in the US.

          And when times get tough, these H-1's, illegal immigrants, and other special practices are rarely scaled back, and so we end up with unemployed ci
          • Re:What's better? (Score:3, Insightful)

            by vlad_petric ( 94134 )
            The H-1 part I simply don't see. It's simply bringing people that are (much) more skilled than the average American. Furthermore, there's a cap on them - about 200k IIRC. This means that there's basically one H-1 worker per 1500 Americans. Finally, let's not forget that other countries are sometimes better in training specialists in some fields. It's very weird that these countries are very upset that they're losing their top specialists (after subsidizing their education), while in the States people feel t
            • It would be different if the H-1 imports got the same salary rates as everyone else, which they generally don't. An indian programmer is going to make a lot less money then a native one. So every single one of those 200K people is going to go straight to the workforce - and the locals either take 40% pay cuts or find a new profession.

              There's not many professional professions where this type of thing happens. Think doctors, lawyers, stamped surveyors, etc. Good programmers and IT professionals usual
      • Basically what you are proposing is a division between the industrialized economies with higher living standards and the developing economies with large labor supply. Closing markets strategically will make things more equitable, hence free? This is a foreign idea to free trade or free market theory.

        Closing a market to supply (or demand) makes it "non-free", correct? We are talking about liberal (to neoclassical) economics right?

        The U.S. market is purchasing the labor of India, China, and Mexico. Foll
        • Actually, there's a solid argument for stopping the free flow to countries like India. They've got incredibly high tariffs on imports while benefiting from the rest of the world's relatively open markets. It would make sense to recipricate tariff reductions, and punish those who sway before local producers. This is one of the goals of the WTO and it's how we get to "liberalized markets" that ultimately benefit consumers in both markets. The side effect of taxing their own base like this is a reduced demand
      • Allow me to explain. A shortage of labor is a normal market force, and government should not intervene to counteract this force. Two of the effects of a shortage is (1) to boost wages and (2) improve working conditions.

        The first problem with your arguement is definition of what constitutes the "market." Just like in thermo when you can arbitrarily define what constitutes a system, in economics you can do the same when it comes to a market.
        If you define the labor market as all the workers in the US then a


      • Goddammit MOD PARENT UP!!! It's been a while since I read something as true and as insightful on slashdot. And modders beware particularly of those who disagree with him in long posts; writing too much does *not* equal being insightful - I say this because I have just seen a long post of someone who disagreed with him modded insightful though on reading it I had the certain feeling the modder either was an idiot or did *not* bother read that reply and examine its content for vailidity.

        And for anyone who
      • Making it more difficult for the US to import labor (H-1b visas) is counterproductive. Overall, the US is currently a net *exporter* of services. Your proposal would actually reduce US exports, since it leaves those resources in other countries. Then we don't get tax revenues, etc. from them.

        "Free trade is good -- only when we are trading with other societies that maintain (relatively) free markets."

        This is simply not true. Free trade is very good to the US for two reasons:

        1. It allows us to trade g
      • by Courageous ( 228506 ) on Sunday July 10, 2005 @09:33AM (#13026119)
        A shortage of labor is a normal market force, and government should not intervene to counteract this force.

        Well, except for the fact that you have it all absolutely backwards! When the shortage appears, the government, which is already intervening elects to relax its interventions.

        A limited number of foreign workers (fixed number of H1-B's) is itself per se an intervention. Without this intervention, the market would freely correct itself, through unrestricted immigration.

        So what we have is a market where the government defacto creates shortages (through dejure immigration controls), but occasionally lets up on the shortage-creating phenonomenon, allowing normal market dynamics to function.

        C//
      • by VP ( 32928 )
        While the government raised the number of H1B visas in response to the Internet bubble (belated as usual), the H1B numbers have since shrunk back to pre-1995 levels. All H1B visas for 2005 were exhausted in a single day last October. According to the parent, since there is a cap on H1Bs we should be seeing the raising wages and improved working conditions. I can't wait...
        • However, L-1 visas were substantially expanded during this period--and there are various ways to use L-1 workers for the same purpose as H-1b workers. Also, a substantial number of H-1b workers are and were exempt from that cap. Also the 1998-2002 expansion was _so_ huge that it created a huge backlog.
  • Dammit! (Score:5, Funny)

    by dancingmad ( 128588 ) on Saturday July 09, 2005 @08:32PM (#13023528)
    A bot taking another IT related job! Where will it end?
    • A bot taking another IT related job! Where will it end?

      With the Cylons.
    • We're all doomed! One day soon, computers will be building themselves, and running everything. In fact, they'll even be posting to /. with even more insightful comments than ours!! They may even achieve sentinent life and not realize they're robots! Wouldn't that be scary?

      -- Message posted by Slashbot #1192392, activation date: 06/23/2004.
  • But how many jobs have he gotten with his kiddie script monitoring one website? There's no alternative to updating your resume, prowling multiple websites for job listings, submitting your resume, and playing phone message tag until you land a job (or, more likely, a contract).

    As for my experience looking for a job recently in Silicon Valley, I would say things are getting better for contract work. From what my friends are telling me, companies don't seem to be hiring for the long-term. I even got an ema
    • But how many jobs have he gotten with his kiddie script monitoring one website? There's no alternative to updating your resume, prowling multiple websites for job listings, submitting your resume, and playing phone message tag until you land a job (or, more likely, a contract).

      You don't understand. This is slashdot. We form burning urges to automate drudgery instead of live it. A true geek can only pull so many fake smiles and handshakes before going insane. When cornered, skunks spray; geeks code.
  • Could you at least have chosen colors other than red and green for the first two choices!? I'm colorblind, you insensitive clod!
  • by King_TJ ( 85913 ) on Saturday July 09, 2005 @08:48PM (#13023588) Journal
    One question though: Why computerjobs.com? I'm not real familiar with their site, but are they one of the sites that claims to consolidate complete listings of I.T. jobs from a number of other large job search sites (Monster, CareerBuilder, HotJobs, BrainBuzz, etc. etc.)?

    If they really do get a pretty good number of I.T. related listings all collected up in one place, then yes - I think this is a pretty useful little graph/tool.

    I've been out of work since the beginning of May, and living in the St. Louis area, it seems to me that there are currently very slim pickings. I keep hearing talk of the economic recovery, but at least around here - I'm not really seeing it.

    According to your chart, that would be an accurate accessment too - since it clearly shows a sharp decline in I.T. jobs available in St. Louis since April of 2005. (And worse yet, I'm really mainly interested in the hardware side of things, but if you look at that specifically - you see that in my city, there were only a grand total of about 2 jobs fitting that category, at any given time!) In the whole U.S., it looked like I.T. hardware jobs only averaged around 1,200 *total*, for that matter. Not good... not good at all!
    • "One question though: Why computerjobs.com?"

      Personally, this submission to Slashdot just looks like a cheap way of advertising a lame job board. If there was a real intent to track the IT market, all one would have to do is go to dice.com, and use the bots that you get for FREE there.

      Dice.com is bigger, and nationwide. So you could really track how the IT market is doing over time.

      Dice is also the only board which takes consultants seriously. That is, it offers specialized selections to make it easy

  • by Umbral Blot ( 737704 ) on Saturday July 09, 2005 @08:54PM (#13023615) Homepage
    You know if you want to check out how tech jobs are doing why not go here [mshiltonj.com]?
  • >> Should I study Data Warehousing or E-Commerce

    You should study what you enjoy to study and work on. If this means a less trendy job, so be it. I'd rather eat broken glass than do web development, for example. Even if it was the hottest job on the market, I wouldn't do it anyway.
  • Jerks (Score:1, Interesting)

    by Anonymous Coward
    For all the liberal bleading heart crap I read on this site, there are some mean people here. I thought the information was fairly interesting.

    Read his resume, he's in Atlanta. I went to Georgia Tech and lived in Atlanta for a number of years, and this site was the first I would check when job hunting. It's fairly big there. They don't aggregrate other site's jobs, employers have to pay. With the exception of the head hunters, they were quality jobs. (But I've found that every head hunter posted job I
    • I was 18 months between 'real' jobs, and three years between jobs I really enjoy(ed). Several times in there I heard/read things like "You have to remember many people in IT just got in for the boom, 'good' people are still in demand'" which of course means if you're not in demand, you're not good. Pretty depressing. Good thing I could look back on 15 years of very happy employers to know it wasn't true.

      Thank intelligent design for the various projects and classes helped keep me sane, and 'kept my skil

  • Whats that (Visa Sponsor)?
  • ...I would be pleased to get a line on just ONE "hot IT job". Write me a bot that will do that, and suddenly I'll care about your war3z. I live in Japan though. Interesting position. Or, lack thereof.
  • Timothy now has all power-ups and 30 continues, ICT market unchanged. Konami executives currently giving no comments.
  • where the f is software on the skill list??
  • by jkeegan ( 35099 )
    I worked at CMGi for five years, and a good portion of that was working for a daughter company named InfoMation on a project called Echo. It was a web scraping tool that let you create your own personal newspaper, from any source you wanted (web, usenet, mail, rss feeds, etc). Imagine google alerts but you could create custom news feeds to any sites you want.. Have it look each day for new albums from your favorite bands by creating custom searches on music sites.. Have it monitor your competition's websi
    • Re:Man, I miss Echo (Score:2, Interesting)

      by Ian.Waring ( 591380 )
      Unfortunately our sales department tried selling it to the insurance industry, which takes way too long to purchase technology (as opposed to, oh, say, a stock broker, who would want to follow every stock he's got his clients invested in.. go figure).

      And yet if my stats are correct (I work for one of the largest IT resellers in Europe), the Insurance Industry are the #1 early adopters for virtualisation software. I think something like 18 of our top 20 VMware customers are all Insurance or Financial Servi

  • What to study? (Score:2, Insightful)

    by Samir Gupta ( 623651 )
    Should I study Data Warehousing or E-Commerce?

    You should be studying Computer Science...

    Ever wonder what happened to all those mainframe or COBOL folks? Knowing about E-commerce, Unix, Windows, Java, XML, or whatever the technology or trend du jour is might be impressive now, but in a few years, come the next thing, where will you be then? These things change at the blink of an eye.

    On the other hand, algorithms, computability theory, formal languages, predicate logic, etc. don't.

    A solid foundation of

    • I would agree that it's good to have the theoretical background, but without the desire to learn new things, I'd say it's much less useful than following the trend du jour.
      • Re:That's a myth (Score:2, Informative)

        by sedyn ( 880034 )
        I'm a CS major, and frankly, it's disgusting to see how many people can get a degree and not know how to program at all.

        That being said, a language is nothing more than a way to describe a concept. Ask the "trend du jour" people about programming concepts, and you'll probably get a bunch of software engineering babble in the reply. (an experiment of this would be interesting, given that the person wasn't mislead)

        My basic belief in learning computer science is to learn how NOT to be a code-monkey. Any i
  • Obviously (Score:4, Funny)

    by eclectro ( 227083 ) on Saturday July 09, 2005 @09:38PM (#13023792)
    Should I study Data Warehousing or E-Commerce?

    You should learn how to provide vertically integrated e-commerce solutions providing dynamic interaction to customers in synergistic markets.

    Knowing how to work a sock puppet [tvacres.com] also helps.
  • Sixth item from the bottom of the list.

    check it out

    http://www.thinkgeek.com/gadgets/electronic/732d/d etail/ [thinkgeek.com]

  • If you're in the UK... http://www.jobstats.co.uk [jobstats.co.uk]
  • legacy systems? (Score:3, Insightful)

    by spoonyfork ( 23307 ) <spoonyfork AT gmail DOT com> on Saturday July 09, 2005 @09:57PM (#13023859) Journal
    The chart has an option for "Legacy Systems" which sounds way too general [wikipedia.org]. I mean, isn't everything currently running in production legacy?
  • Most jobs arent even posted... So while you may get an estimate, its no where accurate enough to stake your future on.
    • Tell me about it. Here in LA I have been getting calls from recruiters for the past 3 months. And these calls are for my expired DICE resume or other resumes I havn't updated in 6-12 months. If only my CCNP had experience to go with it I'd have doubled my pay rate by now. Instead I'm still trudging along doing MCSE type work (plenty of experience there, just all too much of it in small enviornments, otherwise I'd have doubled my payrate again!)

      I just talked to some recruiters yesterday in downtown LA, and
      • Isn't that a paradox though? I mean, part 2. If you want 2-5 years experiance with X, doesn't that mean that X likely will have needed to be around for 4-10 years for an average of people to get experiance with it? And in IT, something that is 4 years old might be outdated...

        I mean, even now, I'd guess you'd just start to see people with 3 years of experiance in Win2k in large amounts (as most places don't deploy stuff right away) yet Win2k is considered rather outdated...

        I mean, sure I have 3 years of ex
  • Ted Shieh was doing something like this up until the job crash [liquidmarkets.com] but then he terminated his data gathering.

    Too bad -- getting a profile with the same measurement instrument over the entire bubble-bust cycle would have been very valuable for historians of technology and the politics of H-1b visas most particularly.

    I suspect what the graphs would have shown was a far faster drop-off in the jobs for languages other than Java than for Java due primarily to the fact that the Indian CS diploma mills were set u

  • I recently read a blog post by a VC who took a different approach to predicting the future employment rates in the U.S. As a VC, he looked at VC investment in the past N months in the U.S., looked at how "early" those investments were, and basically concluded that since there was a lot of investing, these companies will be growing and hiring people. I don't know how big of a drop in the sea of employment this is, but I thought it was an interesting way of analyzing the situation.
  • Wasn't there another guy who used to do something like this? Called it something like SkillsMarket?

    He used to link to it in his /. sig but then, about half a year ago, he announced that he wanted to "move on with his life" and would sell his code, site and related goodwill.

    I think part of the deal was that he was going to open the source but I never heard any more about it. Anyone know if that happened? Or of any FOSS projects working on graphing employment data?

    • Wasn't there another guy who used to do something like this? Called it something like SkillsMarket?

      He used to link to it in his /. sig but then, about half a year ago, he announced that he wanted to "move on with his life" and would sell his code, site and related goodwill.

      I think part of the deal was that he was going to open the source but I never heard any more about it. Anyone know if that happened?

      SkillsMarket [mshiltonj.com] is apparently still in business, and Hilton is still trying to sell it.

  • And for a used car... Listings on major newspapers' web-sites are auto-generated from databases and so are uniform and thus easily parsable...

    My script would poll the site every hour or so and notify me, if anything not seen before appeared.


  • I'm sorry, but basing the "health of the IT employment market" on the number of job ads is flawed.

    Employment consultants are somewhere just below Real estate agents and car salesmen, where their job is not just making an individual sale(read commission), but portraying a positive and bouyant market, while continuing the industry's success.

    By glutting the market with jobs ads that many would not stand up to audit, achieves several things:
    - A percieved shortage means that commissions remain higher.
  • by heroine ( 1220 ) on Sunday July 10, 2005 @04:11AM (#13025218) Homepage
    Now one thing you learn after college a lot more than you learn in college is exactly how to differentiate between jobs. The real world isn't defined as much by the type of programming you do as much as the scope of your responsibility.

    Resume readers don't care if you're a Windows programmer, a UNIX programmer, a hardware designer, or a secretary. They want to see if you're a programmer, project lead, project manager, marketing manager, director, etc.

    Things like Google, open source, wiki have leveled the playing field to where it doesn't matter if you study hardware, windows, AS/400, or UNIX. These things can all be learned by anyone at any time. In modern companies the skills at any given level of responsibility are being learned on demand as they're needed. Hardware designers one day are being used as UNIX programmers the next day.

    Todays differentiation is in how much responsibility you're capable of having. Most resumes are being divided into management, sales and programming and as far as we can tell from the 36 checkboxes, management is the place to be.

  • Nothing new (Score:4, Informative)

    by StrawberryFrog ( 67065 ) on Sunday July 10, 2005 @05:43AM (#13025428) Homepage Journal
    Jobstats.co.uk [jobstats.co.uk] has been doing this for years, and aggregating counts of listings from multiple sites.

New York... when civilization falls apart, remember, we were way ahead of you. - David Letterman

Working...