Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Bug IT

How Do You Get Better Bug Reports From Users? 205

Posted by samzenpus from the that's-a-feature dept.
itwbennett writes "You can try to train them, you can try to streamline or automate the process, you can demand that all bug reports go through a middleman (i.e., a QA tester) or you can throw up your hands and accept that users will forever submit bug reports that in no way help you solve the problem. Like the stages of grief, you've probably tried or experienced all of these at some point. But have you found any approach that really works for getting useful bug reports from your users?"
This discussion has been archived. No new comments can be posted.

How Do You Get Better Bug Reports From Users? More

How Do You Get Better Bug Reports From Users?

Comments Filter:

  • Make them feel connected. (Score:4, Insightful)

    by MindStalker ( 22827 ) <mindstalker AT gmail DOT com> on Thursday July 11, 2013 @07:16AM (#44248947) Journal

    I think bug reporting was one of the things that the early Mozilla project did right. If anything give your users tools to track down the source of the bugs themselves. Your few power users who are developers, if you give them tools to view the stack trace or other such things often will, and they might be able to do much of the work for you.

    • Our power users are people who are good at selling you stuff you don't actually want. None of them are developers.

    • Re: (Score:3, Insightful)

      by DNS-and-BIND ( 461968 )
      How is a USER supposed to track down a bug? They're users.

      Sounds to me like the developers just didn't like dealing with bugs and wanted the users to do it all and present them with a gift-wrapped solution.

      • Re:Make them feel connected. (Score:5, Informative)

        by Dupple ( 1016592 ) on Thursday July 11, 2013 @07:30AM (#44249051)

        How is a USER supposed to track down a bug?

        In the same way a beta tester does (among other things). Every time I do 'X' then 'Z' will happen instead of 'Y'. Please fix.

        Recreating a repeatable problem is a pretty good start. In my experience the hardest part seems to be getting the average user to explain what the problem in a way that can be understood that is beyond 'it just doesn't work'

        • One of the commonest problems is that the error occurs in a way that's not repeatable from the user's point of view. For example, he's editing a file and the software does something weird. Was it that I pressed the A key after I pressed the ESC key? Was it because I pressed the space bar while moving the mouse? Was it because the program's window was maximized in the right display instead of the left display? Was it because the program auto-saved while I was editing and part of my file had swapped out?

          • Re: (Score:2)

            by Kjella ( 173770 )

            There's an 80/20 ratio here, if you fix the 80% that fail the same every time then you wouldn't solve everything but you'd solve a lot. And sometimes the exact set of conditions is obvious after a few tries (Example: A chess program I use, if you have hit "analyze" but is still reviewing the options and haven't started it, get challenged to a match (usually a rematch) and hit cancel in the analysis dialog, the client crashes. First time I thought random crash, second time waaaait didn't it crash before on s

      • Re: (Score:2)

        by khallow ( 566160 )

        How is a USER supposed to track down a bug? They're users.

        They're users because they're using the program in question. They run into the use cases that trigger the bug.

        And an open bug reporting system means that users can get educated on what sort of bug reporting actually helps developers.

      • Re: (Score:2)

        by Jawnn ( 445279 )

        How is a USER supposed to track down a bug? They're users.

        Sounds to me like the developers just didn't like dealing with bugs and wanted the users to do it all and present them with a gift-wrapped solution.

        Bullshit.
        Asking for the most basic of useful information, as in "steps to duplicate" is not "gift wrapping". Bugs that appear at the user's hands are, almost always, bugs that developers and alpha testers did not see. It is reasonable, then, to expect the user to do what they can to document the steps that gave rise to the unexpected behavior. It is absolutely not reasonable to expect the developers to cast about blindly, guessing at how the user might have behaved.

  • No. Well, not yet, but I don't expect that to change anytime soon.

  • Force them to be better (Score:2, Insightful)

    by Anonymous Coward

    We have a big internal web app, when it throws an error it will log that error as part of a graceful error handling process. (Page, scope vars, user info, time, etc) From there, the reporting system requires a unique bug tracking number (the error number logged above) and it checks for uniqueness and to see if it exists before it lets you open it. The only edge case it doesn't catch is when our tracking mechanism is also broken and these tend to be severe outages that are pretty easy to diagnose and repro

    • Force them to be better. By beating the required information out of them. They'll soon learn to pay attention to what they did wrong, and to any error messages, if they consequences of not doing so are a thorough thrashing.

      Of course, in some nancy pancy places they frown on beating the users. In those cases, I guess you could try the carrot approach. Give them a chocolate coin for every useful detail in the error report.

      Have a clear list of what is required and what is helpful for any error report. Then, ma

  • Period ( . )

  • Depends on the user base (Score:4, Informative)

    by Enry ( 630 ) <enry@@@wayga...net> on Thursday July 11, 2013 @07:24AM (#44249009) Journal

    I've worked in support organizations for 15 years. In a commercial environment where you can afford the staff, having a tiered approach works best - you have a help desk to gather and refine the questions and answer the small stuff, then work your way up to the engineers that wrote the code. The tough part of that is having a skilled enough help desk to know when to skip the canned questions and just forward a request on once you have the right information.

    For organizations without those resources, you need to rely on the user base to be the help desk. Give them as much concise information as possible and frame the bug submission so that any and all needed data is in the report. Then it's up to the developer to give good information back to the user.

    As an example, I had a problem with my laptop's trackpad going wonky. Ubuntu made it really easy to compile information about my system and submit it as a bug report, then open it for me so I can add any additional text I wanted. The answer I got back asked me to try a different kernel, and included well-documented links and information on how to get and install it. Just saying something like "yeah, go grab something out of backports" doesn't help the user if they have no idea what you're talking about.

  • Depends on your users (Score:5, Insightful)

    by dkleinsc ( 563838 ) on Thursday July 11, 2013 @07:27AM (#44249023) Homepage

    If your users are, say, someone who works in a different department of the same company as you do, then I've found that the best approach is to positively encourage them to send me reports. If I can't figure it out from the report, I'll even walk over and ask them to show me exactly what they did to cause the bug. When I do that, I give them some tips on how to write a report that explains that more clearly: "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."

    If your users are mostly developers, then by all means give them easy access to stack traces, memory dumps, etc.

    If your users are mass-market end users, then include with your application a system that tracks user actions and sends it (along with a crash dump) to you at the user's request if there's a problem.

    If your users are calling in to your company to complain, plan on using some kind of desktop sharing software so that the user and you can see exactly what happened.

    If your users are dumber than a pack of hammers, then I have no answers for you.

    • " "I logged into the website, clicked 'Foo', entered 'baz' into this box, then clicked 'Submit'. I got an error page instead of this other page I was expecting."

      I see your problem already. You should have type "bar", not "baz"

  • Worse error messages (Score:5, Interesting)

    by sheetzam ( 454981 ) on Thursday July 11, 2013 @07:31AM (#44249059) Homepage

    When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.

    • Re:Worse error messages (Score:4, Insightful)

      by Rob the Bold ( 788862 ) on Thursday July 11, 2013 @07:50AM (#44249191)

      When writing error messages, don't have it spit out something sensible. Have it spit out something completely crazy but memorable, which you can then grep the code for. Something along the lines of "The Cake has hit the Fan" or "The Chickens are eating Pie". This improves the odds the user will remember it and report it correctly, giving you some hope of finding where the bug is in the code.

      Or make it copy-and-pastable so no human memory is required. Or log it and make sending the log easy enough for your users. A nice user message followed by a more detailed procedure unwind or whatever applies in your application can be mighty useful.

    • Purple Monkey Dishwasher

    • Interesting idea, I like it. The problem I've seen with errors is that they are sensible to programmers but cryptic and meaningless to users. If you are writing errors, put them in plain English (or whatever language you're programming for).

  • Speak to the Submitter (Score:4, Interesting)

    by Capt.Albatross ( 1301561 ) on Thursday July 11, 2013 @07:31AM (#44249067)

    The sooner, the better. You need an agile bug response process.

  • Bug template (Score:4, Insightful)

    by Anonymous Coward on Thursday July 11, 2013 @07:33AM (#44249083)

    In your bug tracking system, provide a bug template they need to fill in.

    Encountered situation:
    Expected situation:
    Error messages, if any:
    Steps to reproduce (please provide screenshots if appropriate):

    If the template is not fully filled out, simply throw it out with "insufficient information/cannot reproduce". Eventually your users will grow up.
    Also, there's nothing wrong with walking up to a user's desk (if physically possible!) and ask them to show you what they did.

  • The BUG! Button (Score:5, Interesting)

    by AndyCanfield ( 700565 ) <andycanfield&yandex,com> on Thursday July 11, 2013 @07:36AM (#44249109) Homepage

    On every page of the site that I did for this company, in the upper right corner, there is a button labelled "BUG!". Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in. The dialog box has a "Submit" button.

    What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically.

    And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page.

    The BUG! button was invented - by me - as a reaction to Bugzilla, which seems to be designed to keep users from reporting problems. I can ignore pointless bug reports, but I want to hear everything.

    • Re: (Score:2)

      by Qzukk ( 229616 )

      Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in.

      "It's not working."

      • Click on it and a dialog box comes up that says simply "What's wrong with this page?" and has a big blank area to fill in.

        "It's not working."

        "What is not visible is that the JavaScript code for the BUG! button is grabbing all the information it can from the browser itself - what the current URL is, all the global JavaScript variables, name of the current logged-in user, time and date, browser type, web page contents, etc. All grabbed automatically. And all stored in a special file on the server. The only thing the user has to tell me is what she doesn't like about that page."

    • Bug reporting has to be extremely easy for users, or they just won't do it. Bugzilla is an absolutely excellent, unsurpassed example of how to prevent users from reporting bugs in software. It is absolutely wretched and damn near unusable.

      Your idea is excellent: let users complain in their own words, but have the software transparently gather all the runtime information needed to reproduce and identify the problem. That should also work for desktop apps written in Java or C#, which is something I think I

    • Nice idea !

      In my case, when the program catches an error, it displays a text input and a Send button.

      I expressedly ask the users to explain how they reached this message, and I encourage them to report bugs in order to improve the quality of the program.

      When they click Send, it sends a mail containing their message along with the (hidden) callstack and the error message.

      Most of the time, people are unable to explain how they encountered this error, but at least they always press Send !

    • The only thing the user has to tell me is what she doesn't like about that page.

      And odds are that'll be, "It doesn't work."

  • Show the reporters of bugs you care and appreciate (Score:4, Insightful)

    by Rooked_One ( 591287 ) on Thursday July 11, 2013 @07:42AM (#44249141) Journal
    ... their efforts to reach out to. They are actively trying to help you make your program better. I am not a programmer, however growing up in the pre-nintendo tech days, I know what a bug looks like. I've reported many bugs, and the only response was from Charlie on the Natural Selection (Half-Life mod) team. As much help as he had been in the past when I lost my account info and helped me retrieve it when he was just a one man team, basically, the response back was lacking. I chalked it up to being busy.

    If you're telling me you have time to program, but can't take the time to read an email that's bullet-pointed with very specific details and at least respond like a grateful human, then I am just going to say you have way to many e-mails to respond to or you're lying. In the prior, get more help. In the latter, please seek help.
  • No matter what process you employ it will fail, what you need is someone who is willing to follow-up with the users and step in their shoes, I have realize users are always willing to report a bug as long as it makes them feel important and FOLLOWUP.
  • I believe it was Google who at some point had an option to highlight the portion of the page that showed the problem, as well as submit a comment about it. Being able to see exactly what this user is talking about would help quite a bit when I go to fix a bug.
  • First stage is a php form I wrote which asks them very directed questions. They have to answer questions like "What was running on the computer when the bug occurred. Were you running the software in VM, Did you follow the setup directions properly, Did you download the newest version from the test server etc....

    Second stage is they let the form take logs from the computer, I automated the process so the form takes what it needs. They literally hit start and go have coffee or something well it works

    • Did you follow the setup directions properly, Did you download the newest version from the test server

      I hate when QA tester ask bad question and complain about useless answers.

      sigh

      • Should of proof read that a little better.

        • I wasn't responding to the dropped plural and incorrect capitalisation. I was responding to the irony in you complaining about bad questions and the achingly bad questions you posted as examples of what you ask.

          Asking the user, "Did you download the newest version from the test server", when you already auto-grab user information, which I would damn well hope includes build info.

          Or ever asking anyone "Did you follow the setup directions properly?" No, I sincerely believe I'm the problem, that's why I'm subm

  • Have tried everything (Score:3)

    by Stormcrow309 ( 590240 ) on Thursday July 11, 2013 @07:48AM (#44249177) Journal

    Hello everyone,

    I have tried a bunch of ways. Trained the 'expert' users in the area on how to put in a better ticket. Sent tickets back to them because of lack of information. Judicious of cattle prods and a tack hammer... However, users will use what method is easiest to them, which tends to be:

    1. Calling someone they know directly
    2. Emailing someone they know directly
    3. Emailing the ticket capture email address with 'Call me'
    4. Calling the service desk
    5. Screaming at someone from IS in the hallway
    6. Emailing the ticket capture email address with a long email chain which tangentally mentions the issue somewhere in the middle
    7. Complaining to coworkers
    8. not doing anything
    9. Log into the ticket system and put in 'call me'

  • If your organization is small enough, send screenshots together with the report, or if you have some kind of desktop sharing ability (remote assistance, Lync), take your dev through the process of how to trigger the bug. Just had an issue yesterday where they couldn't reproduce it, so we fired up a screen sharing process, they had me load up the Firefox debug console, and I went through the steps... turned out to be my mouse having a higher dpi than what they were using, so instead of getting floats, my sy

  • Security bugs in chrome, firefox etc get actively hunted down because of the explicit cash rewards, it clearly works.

    I recently "submitted a bug" in an IOS app for a restaurant (web contact form, no email, in-app submission etc), they said I'd get a "free stamp" (collect 5, get a free meal etc) if I sent them the crash logs. Not a big deal, (I'm a developer so not difficult for me to do) but even a small incentive encourages me to actually DO it.

    If you/parent company can give away something like like loyalt

  • I put a button in my software. Click it and a form pops up with a "describe what you were doing when a crash or bug happened, or make a suggestion for a new feature" type message. It also, behind the scenes, captured the state of the program when the button was pressed. That worked well. I got a bunch of new feature requests, as some good hints as to what the user was trying to do when things went south. YMMV
  • And it's not working!

  • Actually yes... I get very good bug reports from users. Luckily I did tech support for about 15 years before I got into development. So I had a lot of experience dealing with people that basically had no idea what they were doing. Keep in mind, I work in a corporate environment where what I create is consumed by a limited set of about 5k people.

    1. Relationships: Where possible I get to know, personally, the people that will use my software. I can't know them all because there's thousands. But I can find key

  • I am a developer. I write good bug reports. But when using a product I'm not working on, if the devs or the process seem out of touch, I don't tend to. If there's a crash report, I won't invest much because the seems to be zero investment back.

    Whenever a bug is submitted, a real response... not just an automated mailer... should be sent within a day. Get more details if needed, provide an ETA. Otherwise, we're spitting in the wind, and it doesn't seem worth the effort.

  • In one company where I worked, I was able to have the end users funnel all their bug reports through the company's internal education department. If the "bug" were actually a prevalent user error, the education department would note that and modify their internal courses to instruct the users on proper usage. If the"bug" were actually a programming problem, the bug reports we got from the education department were very helpful to resolving the problem.
  • By this I mean that you should instrument the code with real, meaningful activity logging, not just some afterthought that grabs a stack trace and some state variables (although you'll want to have that data, too). If you instrument your code with logging that produces readily human-interpretable information about what's going on, the payback is huge, because it makes internal developers' lives easier, and it allows even first-level support folks to to a better job of triage and analysis. It's really import

  • Long term breeding program (Score:3)

    by mseeger ( 40923 ) on Thursday July 11, 2013 @08:59AM (#44249959)

    I think the only chance is to create your own breed of users... May take some time ;-)

  • Force your users to create accounts at SuperDuperBugTrackingThingy.com to report anything. Then send them daily email updates from said tracking system. Also get your moderators to admonish any user that dares accidentally post an existing bug or posts in the wrong section of their clearly laid out bug tracking hierarchy. Or you could go full retard, like Google, and force people to use Google+ to do anything.

  • If you can interact directly with your users, use Teamviewer or any screen sharing program. Watch them reproduce the error and you'll learn a lot.

    This is closely related to the first point of this other comment [slashdot.org] so even if you can't interact with all your users (too many of them or too many layers inbetween) there will be someone who can. Again, a recorded screen sharing session can make its way to you.

    PS: I know, making the user install the screen sharing program and properly start it can be a pain but many

  • automated test can work or they can pass the test but still fail in ways that automated tests can see.

    Dumping QA on to coders work load does not really work and you don't want the same people who code to be the testers any ways.

    QA people need to be able to think out of the box and have full control over there PC systems for testing as well as well having more then 1 log in as well.

    Also in power your help desk to be able to help and not be held back with call times and other BS.

  • We have a huge problem with vague or misleading error messages. Instead of messages like "A file could not be accessed", how about you tell me which file could not be accessed?

  • Just give them more interesting bugs to report. Link the error reporting to an easter egg for a fun game. Give them bonus points for originality. Before long, users will be lining up with reports on exactly how to reproduce the bug.

  • I am a user, not a developer, so I have a little different perspective.

    First, what does not work for me is a faceless drop box. Without feedback and a sense that the other end cares, it is hard to motivate to put in a bug request, let alone a good one. Submissions to nameless drop boxes become a rant to vent frustration.

    For an external vendor, company A, whose design software I have now been using (suffering through) for 14 years I gave up submitting reports after 3-4 years. I never heard anything back,

  • Shoot the users that submit bad bug reports. Within a few generations, only users that submit good bug reports will remain.

Slashdot Top Deals

Think of it! With VLSI we can pack 100 ENIACs in 1 sq. cm.!

Close