Follow Slashdot blog updates by subscribing to our blog RSS feed


Forgot your password?
Security IT

Distributed, Low-Intensity Botnets 167 writes "We have seen the future of botnets, and it is distributed and low-key. Are sites running free software finally becoming malware targets? It all started with a higher-than-usual number of failed ssh logins at a low-volume site. I think we are seeing the shape of botnets to come, with malware authors doing their early public beta testing during the last few weeks."
This discussion has been archived. No new comments can be posted.

Distributed, Low-Intensity Botnets

Comments Filter:
  • First Post (Score:3, Funny)

    by Luke727 ( 547923 ) on Tuesday December 02, 2008 @06:02PM (#25966827) Homepage Journal

    And I didn't even need to use a botnet!

  • Isn't that what already do? Distributed, rather than monolithic, spamming?

    Anyway, it's not that surprising...

    • Re: (Score:2, Insightful)

      by davester666 ( 731373 )

      Um, ssh attacks aren't new. They've been hitting my server's for year's, and mine are for a private consulting company, with trivial amounts of random 'consumer' traffic.

      • Re:Isn't that... (Score:5, Interesting)

        by Duckie01 ( 10586 ) on Tuesday December 02, 2008 @08:22PM (#25968895)

        Yeah these worms were attacking my home linux router as well, like a year ago or some.

        Worms just tried to brute force ssh using "administrator" and such as username. I guess they were trying to get into badly (default) configured broadband routers. That's never going to work of course on my linux box but all the login attempts caused the hd to be busy *all* the time.

        My sollution was to drop ssh packets by default in the firewall. Not that these attacks were likely to succeed but I didn't want my consumer grade hd to wear down in a year ;) I then created a small php script that'd insert a firewall rule to accept ssh connections from the IP it's called from. Finally I password protected the php script with .htaccess.

        So now I can enable ssh to my machine wherever I am, while still blocking the rest of the internet.

        • Re:Isn't that... (Score:5, Interesting)

          by weetabeex ( 1065032 ) on Tuesday December 02, 2008 @11:04PM (#25970505)

          You could also be interested in port knocking [].

          Turned out to be quite handy when I had that same issue with bots connecting to my ssh port all day long.

        • Have you posted a copy of that script somewhere? I would be interested in checking it out.
          • by Duckie01 ( 10586 )
            Sure there's not a whole lot to check out tho ;-)

            Well initially I just had

            $ip = $_SERVER['REMOTE_ADDR'];
            exec("sudo /sbin/iptables -I INPUT -p tcp -s $ip --dport 22 -j ACCEPT");

            Just now I niced it up a bit taking http proxies into account, report the results and do some error checking (I'm absolutely no php wiz, just grabbed the function from and added the rest... it's all pretty simple):


            function getIP()
            if (isset($_S
          • by Duckie01 ( 10586 )

            btw in /etc/sudoers I use

            www-data localhost = NOPASSWD: /sbin/iptables

            For .htaccess check out this page []

        • This is why my openbsd box is called 'linksys'. LOL. Might as well have some
          fun with a little misdirection. Sure enough people try to log in as admin, but rarely root.

          • by Duckie01 ( 10586 )
            Mine was called "router" I think... hmm I'd guess they're not smarter than just targeting ssh daemons on port 22 tho.
  • Fleas on a dog (Score:5, Insightful)

    by try_anything ( 880404 ) on Tuesday December 02, 2008 @06:09PM (#25966949)

    If the bad guys can siphon off what they need without being more than a mild annoyance, they can operate without fear of retribution.

    • by Nutria ( 679911 )

      Fleas on a dog ... without fear of retribution

      Flea deterrence is a big business (at least in the USA).

  • by knarf ( 34928 ) on Tuesday December 02, 2008 @06:14PM (#25967035) Homepage

    I've seen SSH probes on my one-man-and-a-dog site for aeons. I don't think there's anything out of the ordinary, the scum has been trying (and failing) to get in for as long as I've had something listening on the 'net - and that is a long time. There's also nothing new in them trying to root FLOSS-sites as those sites - with their fixed IP addresses, good uptime, high reliability and abundance of crappy PHP-scripts to open the doors - make for good C&C hosts for their flock.

    So all I read from this flog is that a grumpy BSD user should probably check his logs more often. This is nothing new.

    • Yeah, I had flashbacks to 2000. I remember setting something up (I think it was portsentry) that would put ip's in the hosts.deny file after too many bad logins, or was it hitting too many different ports on my PC? (probably both)

    • For real. I had my single-page-o'-html site up for a little more than a week with inbound 22/80 open before getting ssh probes. Heck, even when I had my dyndns domain, it was getting probed.

      Nothing new at all.

    • Precisely my thoughts on the matter. The handful of low bandwidth sites I've worked with that had SSH would have endless logs of failed attempts. And it seemed a large percentage of which even came from within the data center that hosted the servers (probably a sign we should've changed hosts, but that wasn't my call).
    • One of the effective ways to not worry about these probes, is to run your ssh daemon on a non standard port.

      So, instead of it being on port 22, run it on port 1022 or some other port.

      You can do that by modifying your /etc/ssh/sshd_config to contain the line: Port 1022.

      This means that scp and ssh will have to be told about the port on the command line though.

      • One major annoyance with the non-standard port is the port flag option passed to both SSH and SCP.

        For ssh: ssh -p2222 ...
        For scp: scp -P2222 ...

    • by daybot ( 911557 ) *

      I've seen SSH probes on my one-man-and-a-dog site for aeons. I don't think there's anything out of the ordinary, the scum has been trying (and failing) to get in for as long as I've had something listening on the 'net - and that is a long time.

      Ditto. I saw a suspiciously high number of genuine accounts being attempted. I drew the conclusion that this was coming from bulk email lists.

      1. Buy bulk email list, containing and others.
      2. SSH to as fred - try some common passwords. Maybe try for good measure.
      3. Profit!

  • Like, cancer, but on computers. In computers. Swarming through an incomprehensibly convoluted and complicated array of computers. Why, oh why, did I ever, start, using, computers?
  • by idontgno ( 624372 ) on Tuesday December 02, 2008 @06:19PM (#25967109) Journal
    the difference between parasitism [] and commensal symbiosis []. Great. It's already hard enough to keep infestation under control in the network ecosystem. When there's no visible damaging impact, how will we detect them?
    • Heuristics. They're an ancient art, but they can work.

      Also, default deny. It's annoying to answer the question, "Do you want to connect to" but it's more annoying not to be asked. If your users have no legitimate reason to be accessing .cn, .ru or other .?? addresses there's no reason to configure your DNS to give legitimate returns for those as an organization. Likewise for the more vile quadrants of the IP space. For some networks it's better just not to ask.

      Sure, it causes a Bal

  • I get somewhere between twenty and thirty attempts per day agianst my SSH server alone. The server blocks the IP permanently after 3 bad attempts and they always try repeatedly until blocked. most of the attempts come from cable or dsl address spaces. I use gibberish for usernames and only allow certificate based authentication. They still keep trying however.

    • by cdrudge ( 68377 )

      I'm in pretty much the same boat with my home ssh server, except I would get 20-30 attempts an hour. Occasionally I would sift through some of the attempts just to see what they were trying to guess as usernames. Root and admin were usual attempts, but I would also see oracle, sa, mysql, etc as well as just random first names like John, Mary, Bob, etc. I eventually grew tired of it just filling up my log files so I switched to a non-standard port and haven't seen an attempt since.

      • I eventually grew tired of it just filling up my log files so I switched to a non-standard port and haven't seen an attempt since.

        I keep ssh open on the standard port because I block offending hosts in pf. It is much easier to detect attacks on ssh, but I get attacked on other ports as well, such as pop3 and smtp.

        I would rather know who the enemy is, than be completely blind.

        If anybody is interested I would be happy to post my current list of blocked hosts.

    • by ShaunC ( 203807 ) on Tuesday December 02, 2008 @06:48PM (#25967561)

      Yeah, same here, except right now there's a rather humongous distributed bruteforce campaign going on. The 20-30 attempts I tend to see have skyrocketed to several thousand per day. It's actually pretty impressive - it's clearly a distributed sequential dictionary attack. Most of the IPs will only try once or twice, in an effort to avoid exactly the sort of reactive firewalling you mention.

      Dec 1 11:17:57 shaunc sshd[35178]: Failed unknown for illegal user griffin from port 20893 ssh2
      Dec 1 11:18:17 shaunc sshd[35262]: Failed unknown for illegal user griffith from port 40689 ssh2
      Dec 1 11:18:30 shaunc sshd[35308]: Failed unknown for illegal user griffith from port 60822 ssh2
      Dec 1 11:18:33 shaunc sshd[35354]: Failed unknown for illegal user grizelda from port 60602 ssh2

      Many thousands of these, seconds apart, all day long. It got so bad that for the time being I've moved sshd to a different port.

      • by dargaud ( 518470 )
        How do I get more detailed sshd logs like you do ? Currently it generates nothing useful (only crash reports). The -d option states that "The server sends verbose debug output to the system log, and does not put itself in the background. The server also will not fork and will only process one connection. This option is only intended for debugging for the server." so it's not helping.
    • Re: (Score:3, Informative)

      by DaveAtFraud ( 460127 )

      You can cut down on the noise by just moving your ssh port to something other than port 22. Such a move won't stop a serious cracker who will do a port scan, etc. However, since it seems to be sufficient to keep the script kiddies and similar types from doing the sort of stuff described in the article, it means you're far more likely to notice when someone with a little higher skill level tries to crack in.

      Best bet if you have a small user base is to only allow public key authentication and move the ssh p

  • by girlintraining ( 1395911 ) on Tuesday December 02, 2008 @06:23PM (#25967171)

    Okay, how is this different than previous patterns of hacking activity, other than the fact that they're aquiring compromised machines via a bot net? It's not! These "security researchers" remind me sometimes of my pothead friends. You can always tell someone who's new to smoking weed because they constantly ask the question, "but have you done it on WEED?" It's like somehow the idea that these people are using a botnet makes it all strange and new again. No, fail!

    • by ShaunC ( 203807 ) on Tuesday December 02, 2008 @10:35PM (#25970231)

      Okay, how is this different than previous patterns of hacking activity, other than the fact that they're aquiring compromised machines via a bot net?

      You're sort of missing the point, I think, in that what's different about this pattern of activity is precisely the fact that it's being done with a botnet.

      For one thing, there's a new level sophistication, primarily in that this bruteforce campaign is not the least bit random. I'm being hit by thousands of distinct attackers, yet the progression of usernames being attempted is undeniably alphabetical. Occasionally a particular username is attempted more than once, but it's typically sequential. One attempt per username with the attacking hosts only making one attempt every few hours.

      The level of coordination required for this sort of attack is unprecedented. Across thousands of bots, each one at any given moment is able to determine:

      • That I am among the pool of targets to be probed
      • That I am, at this precise second, the next target to be probed
      • That this particular bot hasn't probed me recently and is now eligible to probe me again
      • Which usernames have already been probed on my machine
      • The next username, in sequence, that should be attempted on my machine

      In the past, brute force SSH attacks have always been obvious. Typical hit and runs. One host will spew hundreds or thousands of attempts at a target, typically in quick succession, typically focusing on system accounts, and typically trying a shitload of passwords against each account. Firewalls and IDS deployments far and wide will now easily detect (and often block) these attacks immediately because they're so easy to recognize.

      This attack is very different. It's not targeting system accounts, it's hoping to get lucky against a vast list of potential userland lognames. It's only trying once or maybe twice per account. And it's distributing these attempts, round-robin style, across an impressive number of sources, with enough logic so that bot B will not attack host H unless all other bots in the network have sequentially exhausted their "token" attempt on host H.

      What we're seeing is flying under the radar of a shit-ton of IDS/firewall implementations, and is harder to fight.

      I would love to get my hands on the C&C database being used to coordinate all of this. Much as I hate to admit it, the architecture of this attack is unique and innovative, and I'd like to see what makes it tick.

      • by yuna49 ( 905461 )

        It seems to me you could accomplish the same thing without such a complex architecture. I'd build an intrusion robot that iterates over usernames and target IPs and sends each request out via a different compromised host. Imagine setting up TCP proxies or VPN tunnels on each compromised machine that connect it to hundreds of other compromised machines. Then the robot is assigned a target address to which it starts making ssh login attempts via the tunnels. After a while it switches to another target IP

      • by mcrbids ( 148650 )

        I just don't get what the big deal is. Bad? Yes. New? NO!

        I write network applications for a living. This type of coordination with a large number of hosts is precisely the kind of thing that I write professionally. While it takes some work to get this kind of coordination in place, it's probably a man-week with standard tools and commonly available information for somebody who understands how to write network applications.

        Years ago (2002?) I saw similar behavior in what I called a "spam attack". I was the s

  • Go install fail2ban (Score:2, Informative)

    by victim ( 30647 )

    fail2ban will watch your log files and when it sees probing will firewall ban the offender. It has virtually eliminated probing attacks on my networks of machines. Sure, a distributed botnet can still probe you, but I haven't seen that happening.

    Do be careful though...

    • Have two different IPs you can come from. You will eventually ban yourself by being stupid. It took me a year, but I finally banned myself while working on some backup scripts.
    • It is written in python and uses 3M of RAM plus maybe 20M more vir
    • Denyhosts does the same thing, but has a global synchronization method. If one person using the sync feature bans someone, they all do.

      It only works for ssh though. fail2ban is flexible enough to monitor anything and do anything, in response to any pattern. Pretty damned powerful really.

    • by Yath ( 6378 ) on Tuesday December 02, 2008 @06:54PM (#25967657) Journal

      Please read more of the article before posting. The activity being described is a brute-force SSH login attack that is distributed across a botnet.

      (Yes, the title of the article is misleading, as botnets are by definition distributed; the interesting bit is that SSH brute-force attacks against a specific host don't seem to have been distributed before.)

      Here's the relevant bit:

      See for example the attempts to log on as the alias user, 14 attempts are made from 13 different hosts, with only trying more than once. Then thirteen attempts are made for the amanda user, from 13 other hosts.

      fail2ban is not effective against this.

      • by Trogre ( 513942 )

        I am undergoing the same kind of attack here, with what looks like the same dictionary. Fail2ban has picked up some 600 hosts, for about 3000 attempts. So it's not useless for all botnet attacks.

    • There should be a memory efficient alternative, maybe I'll have to write that.

      Mine uses 53 lines of ksh.

    • Actually, a simple one line IPtables rate limit on port 22 new connection attempts (or whichever port you use for SSH is better than Fail2ban or Denyhosts, because it will never lock yourself out.

      I suspect that the author has such a limit on his firewall machine and that it is the reason why he sees the slow attack rates and that the slowness has nothing to do with the attackers!

      • by Sancho ( 17056 ) *

        Most of the time, these limits are by source IP-address. This new attack self-limits the rate of connections from a single IP address, and instead hits the same destination from multiple sources. Rate limiting globally will work here, but will mean that legitimate SSH connections will be slowed down.

    • Re: (Score:3, Informative)

      by ShaunC ( 203807 )

      While it's been pointed out that fail2ban isn't effective against this particular attack, I wanted to point out a similar utility called BruteForceBlocker [].

      It was written as a reactive firewall that parses pf logs on OpenBSD and FreeBSD (pf is "the iptables of BSD"). The coolest feature IMO is that it's a community effort, in that each participating host can elect to share its logs with a centralized server. That server then publishes a list of recently reported SSH attackers [] which you can script into your f

  • by Khopesh ( 112447 ) on Tuesday December 02, 2008 @06:41PM (#25967441) Homepage Journal

    I use Fail2ban [] on all of my iptables-based SSH servers, as it eliminates the possibility of brute-force attacks from single IPs (fail2ban will ban any IP with five failed ssh logins in a ten minute period. The ban vanishes after ten minutes).

    However, this new botnet attack distributes the attack over the IP-space and time. That bypasses fail2ban!

    The only solution I can see to this would be to take an approach similar to the centralized spam-fighting solutions; a DNSBL [] specialized for brute-force botnets. You run something that monitors your logs for failed logins (with a large scope for time, say ten failed attempts in a month). When an IP triggers it, you block that IP for a month and report it to the DNSBL. The DNSBL operates like Spamcop [], trying to verify the nature of the IP (and trying to address the issue), then adding it to the blocklist. Anything listed on a DNSBL gets permanently blocked after one failed authentication, and if your internal list grows too big, any positive IP gets blocked before the login attempt.

    • by Khopesh ( 112447 ) on Tuesday December 02, 2008 @07:11PM (#25967903) Homepage Journal
      I forgot to mention my over-arching solution:
      1. Disable root access: in /etc/ssh/sshd_config, set PermitRootLogin no
      2. Disable password access (require ssh keys!): in /etc/ssh/sshd_config, set PasswordAuthentication no
      3. Use an auto-banning solution like Fail2ban [] to limit attacking traffic and save bandwidth

      SSH keys ensure that you're virtually immune to attacks, since the attack now MUST be brute-force (or break the RSA/DSA algorithms or compromise the server itself rather than an account), and must crack a "password" of over 150 base64 characters representing the 1024 bits of entropy in the key; a completely random printable password of 20 characters has 130 bits of entropy and an 8-word Diceware [] passphrase has only 120; you're not brute-forcing that.

      Preventing root from accessing remotely is just smart (your logs can show who really logged in, your sudo logs show who needed root-level access and when, and you can auto-ban root logins immediately rather than after a set number of failed authentications).

      All we're missing is the extension to #3 to handle distributed attack failures (as proposed by the parent post); with the proper protections, this is a bandwidth issue rather than a security issue (unless you're worried about DDoS, but we're talking about low-intensity attacks here).

      For those of you stuck permitting passwords, you'll want something like John the Ripper [] to brute-force users' insecure passwords before the enemies do. This way you can disable their accounts when you find that they are vulnerable and force them to change their password to something more secure.

      • I don't think root login is the problem here (though I agree most reasonable admins disable it). Those of us with more detailed ssh logging will see that these distributed attacks are not attempting root anyways.

        As someone pointed out earlier, these attempts are usually doing a dictionary attack of common usernames instead. The general MO here is that for any user name from aaaaaaaa to zzzzzzzz from their list, some number of systems (usually 3-5) will attempt that user name, and then the botnet will mo
    • If that product use the recent target, be careful what kernel you use it on. You'll find that after a few weeks the damned target starts matching stuff it shouldn't and you might lock yourself out of your server. IIRC the expiry time winds up becoming infinite.

      Don't worry, it's a documented bug somewhere, and recent kernels aren't affected. 2-year-old kernels are for-sure though. I just can't remember the precise details; I've been bit by it trying to throttle SMTP.

    • The only solution I can see to this would be to take an approach similar to the centralized spam-fighting solutions; a DNSBL [] specialized for brute-force botnets.

      Damn! I just posted about this as a reply to a previous thread, and it definitely belongs here instead.

      Anyway, check out BruteForceBlocker [], it's exactly what you describe, but it's implemented as a plaintext list instead of a DNSBL. Hosts using BruteForceBlocker can report attacks back to the central server. The list of recently reported attackers [] is public.

      It's meant for BSD variants using the pf firewall, but I rolled my own implementation to parse FreeBSD ipfw logs (and report back) in a day or two. A da

      • Re: (Score:3, Informative)

        by Khopesh ( 112447 )
        It appears BruteForceBlocker is a decent BSD implementation of fail2ban. However, it lacks the flexibility; from what I can see, all it does is notice a failed login and forever block the offending IP, which screws users who accidentally forgot to include the ssh key, or who used the wrong user name accidentally.

        Fail2ban actually bans the offending IP after several failed authentications within a small window of time, and the ban is only for a small window of time ... the purpose is not defensive for secur

  • Portknocking (Score:3, Interesting)

    by Pvt_Ryan ( 1102363 ) on Tuesday December 02, 2008 @06:42PM (#25967463)

    Like the OP I was getting loads of hits on port 22. I just setup portknocking and it works a treat.. My other system that I use ssh on (its on the a sub domain of my main site) I just moved to a higher port and that has prevented it from getting the hits..

    Normally I don't recommend Security through obscurity but in the case of automated attacks it is worth while. Just don't rely on it alone.

  • by skeeto ( 1138903 ) on Tuesday December 02, 2008 @06:45PM (#25967509)

    These sorts of attacks are nothing new. If you are running an SSH server directly accessible from the Internet, check your /var/log/auth.log log sometime. You may be alarmed by the surprisingly large number of attacks you get every day, even if it is your home IP with no sort of domain name associated.

    I like to run DenyHosts [] on my machines, which watches this log file and adds suspicious IPs to your /etc/hosts.deny. I generally have several new IPs added daily. Also disable remote root logins, because then the attacker has to guess a username and a password: an extra few bits of security (they try "root", then go on to "tester", "tester1", ... , "guest", "guest1", etc.). And, of course, use strong passwords for SSH accounts. In your logs you will find attackers employing dictionary attacks (which DenyHosts will quickly cut off).

    • Re: (Score:3, Informative)

      by oGMo ( 379 )

      Screw that... I went pubkey-only from any directly-exposed hosts awhile ago. Works great. I keep my Blackberry's MidpSSH key on the hosts, then in an emergency I can log in and add a new key if necessary. Plus if people want an account they can just send in a pubkey and I don't have to communicate a password.

  • Wait until they combine this with Debian's 2^16 weak ssh keys. Do you think that will be easier or harder to crack than passwords?

  • This is not a game changing tactic. My institution has documented these style attacks on several past occaisions. There was some of this going around near the 4th of July. There was an extended bout this time last year. The attackers only use this tactic a few times a year. We have come to expect it on major holidays.

    Economics can not be ignored. This attack must balance reward and risk.

    In a normal SSH password guessing attack, the attacker risks a handful of computers. The committed computers do very noisy

  • I switched to only allowing keys a long time ago. This stops this attack dead, whether it is distributed or not. If I ever lock myself out, or use a computer where I don't have any keys already set up, I have a special, password-encrypted key that I can access over an authenticated https url.

    On my VPS I took advantage of SSH's new configuration features to disable password authentication for outside connections, but allow password for connections coming from within my wide-area private network (openvpn ro

  • I run a FreeBSD box at the end of an ADSL line. Normally I would see a handful of SSH attempts. On a bad day I'd see a couple hundred. This last week, I've seen upwards of 1500 per day, all coming from different IP addresses. It's a straight dictionary attack, moving through in dictionary order. I think I'm in the G's right about now...

    I long ago installed 'bruteblock' on my box, which plonks an IP address for N minutes after X failed attempts (both configurable). It's very small and efficient. But

  • For about the last 2 years I Have been getting continuously hit with botnet hosts every 2 minutes trying every conceivable email address@mydomain. I tried writing a rule to autoblock them with iptables but the list grew into the 10s of thousands and slowed down my machine. I put a greylist on it and now they get greylisted but they keep coming and coming and coming. There must have been more than a million greylisted email attempts over the preceding two years. Far more than the amount of legit mail I ge

  • Pretty lame, people.

    Move sshd to another port?

    Rate limiting?
    IP-based bans on failed logins?
    Elaborate username based bans?
    All reactive solutions.

    A more complex obscurity tactic, but very weak as an extra authentication layer.

    TFA mentions ssh public key auth, and disabling password authentication. That would be much better/more effective than anything mentioned here.

    If you're serious about security, then you aught to actually add another layer. Firewall off ssh completely and require a

    • by Ant P. ( 974313 )

      Firewall off ssh completely and require a VPN connection first

      Just another layer of obscurity.

  • So we are still user password-based SSH authentication?

    • Re: (Score:3, Insightful)

      by ShaunC ( 203807 )

      So we are still user password-based SSH authentication?

      The problem is that in any sort of working environment, where you have a very heterogeneous user base, it's really really hard to enforce anything else.

      Users - even the most basic of users - can be trained to enter a username and a password. They do it on Hotmail, they do it on Google, they do it on MySpace, they're used to the idea that when they want to login somewhere, they have to enter a username and a password. "That's how the internet works." So when their job functions require that they PuTTY into a

  • I was wondering what those hundreds of failed ssh logins were from every day.

    I didn't really have any idea what was up, so I just changed my SSH port to 22222 and the problem went away.

  • I had seen this on my own system back in July [] for the first time, and it eventually went away. It kept up for some time, to the point where I decide to write a little script to watch who is trying to come in.

    Then it came back last month [] and I paid a little more attention to what I had been doing before. There was one significant thing that I did just before it (re)started:

    I placed an ad on craigslist that had a link back to my own server for information on what I was selling.

    We all know that of course the spamming botnets tend to troll craigslist looking for valid email addresses to add to their lists. I would say it appears that the botnets are now looking through craigslist for systems to attack as well.
  • Great, so now script kiddies will be completely unable to penetrate my box, but they will fail miserably in a distributed fashion.

    Do you want to bet that the time it takes to brute force a well secured system in this manner is measured in the thousands of years to millennial range?

Civilization, as we know it, will end sometime this evening. See SYSNOTE tomorrow for more information.