Shortly after I started monitoring servers and networks (roughly 10 years ago) I had a server go down... not ususual. But then another on the site. And another...

When I pulled up the site, I saw that they had 2 circuits and 16 servers and the servers that were already down were split between the circuits. Doesn't look like a network issue. I logged into one of the servers that were still up and checked the event viewer. It was overheating, but not yet to the point where it shut itself down to prevent damage. It's pretty safe to say that the others had overheated as well.

I contacted my front-line rep and asked him to call the site. "Your servers are overheating. Can you take a look in the server room to make sure that everything's ok in there? Verify that the air conditioner is still working and that the fans haven't stopped, or anything else unusual, please." "Oh sure... Yeah, everything looks good. Can you send out a tech to investigate?"

So I dispatch IBM. They get to the site an hour or two later and are escorted to the server room where they find construction going on. They were installing a wall, or something. The carpenters had thoughtfully draped a painter's tarp over the server rack to keep off the sawdust. Unfortunately it also created something of a makeshift oven. IBM tech corrected the issue and explained to the customer what the problem was.

The customer then had the audacity to request that all 16 servers be replaced, just in case there was damage from the overheating.

We complied. 16 new (or possibly refurbished) servers were installed at the site in the next couple of weeks.

Once upon a midnight backup, while my laptop I did pack up,

Finishing a Cisco stack, upstairs I’d finished upgrading.
While I called the Help Desk clearly, telling them my job was nearly
Finished, and I felt sincerely that I home would quickly wing.
I saw a little line a-scrolling, telling tales of packets rolling
Out my NIC, but it was polling no responses to its ping.

Yes it WAS in late October, and my project as a coder, 'twasn't like a souped-up motor

Made me think I’d lost my zing.

Seeing ping loss gave me sorrow, such that I could wish to borrow
Toolset tools highly bizzaro, tools to bring back everything.
For the packets out were ending with no Ack and were not sending

Me to bed while they were rending network woes I dare not sing.

And so I started up my browser, seeing soon a meme of Bowser

Hoped that Sha-Na-Na could wowzer fixes of most everything.

NPM did then reveal the information that would seal

My fate of sleeping without zeal--until I fixed that wayward Ping.
“Where did they go?” I pondered quickly (at 1 a.m. my brain is thickly

Scavenging for thoughts that slickly intuition hopes to cling).

“Back to basics” said the sage—what could I use that would assuage the problem?

Yes!--A bandwidth gauge--would reveal the bandwidth hog.

When applied upon the GBIC, quickly as a football flea-flick, it could tell me

News comedic, and I’d find HIM in the log.

All I had to do was wait and surf on Thwack and contemplate how

Packets soon would not be late, no longer victims to backlog.

Presently a message evil crossed my screen much like a weevil

Gliding just like E. Knievel crossed the Snake minus toupee.
There he was, the slowness-grower, making networks ever slower
Acting innocent, but “knower-NTA” could save the day!

Who would slow his colleagues’ browsing, packet sniffers then arousing

Corporate policy espousing CISSP’s away?

Slippery as collodion, playing Nickelodeon,

‘Twas simply the custodian filling up the T1’s pipe.

Surreptitiously it listened, Netflow shined and then it glistened

Sundry packets as they christened music streams through teletype.

“Will he never stop that streaming?” (I imagined my boss screaming)

“Call the site and start blaspheming!” came his effervescent gripe.

And so between the packets I contrived to setup brackets that would

Put him in straight-jackets, stop his surfing at the door.

NPM and Damework Mini, QoE with footprint skinny,

UDT and its close kinny NTM I did implore.

Packet filters let me study all his traffic from his buddy ‘til

I left his PC bloody, dead and lifeless on the floor.

But I knew he’d soon be calling to the Help Desk, loudly bawling:

“All this slowness is appalling—I am feeling mighty sore!”

So I crafted memo quickly, taking care to not be prickly, showing

Him how network sickly streaming vids we do abhor

Entertainment that’s competing with our business traffic fleeting

Citrix and PC defeating flows that must not hit the floor.

But he never started calling—once his Manager was galling at the

News I shared—appalling!—how he stopped their network flow.

So the process bears repeating, using work for play is cheating, 

And the WAN bandwidth-depleting, every person ought to know.

With the mystery abated, and the WAN bandwidth inflated back to

Normal COS (so weighted), homeward soon I knew I’d go.

“Thanks” (I thought) “for things Orion, and for pizza types Hawaiian!

And for packets swiftly flyin'!  And my favorite Toolset tool.”

Never be without Orion, or your Tool Set, you’ll be dyin’

For some thing to stop your cryin’, and you’ll feel just like a fool.

Now you know the road to glory, and it never need be gory.

I can end my network story, SolarWinds provides the tool.

At a previous job my company was rolling out a new application and our satellite office was experiencing some pretty horrendous delays in the application.   The satellite office was running on a T1.  Once complaints started reaching upper management they wanted answers, quick.  I hop on SW and check out the latency and network utilization.  Good latency and a utilization of 3-5%, definitely not the network.

FIX 1: My manager decides that it is obviously a network bandwidth issue even after I try convincing him otherwise so he orders another T1.  A few days after install, they notice the application is still having the same issue. The same process unfolds, management wants answers, I check SW and see the same amazing stats with good latency and low utilization. 

FIX2: Manager thinks it’s still the network bandwidth so he orders 4 more T1’s to bring the total to 6 T1’s for a screaming 9mpbs.  A couple days after network upgrade the application is still having the same issue. Again the same process unfolds, I check SW and the location still has amazing latency, and they are averaging 1% utilization.  By this point you would have thought they learned their lesson, but no.

Fix3:  Manager still thinks this is a network issue. At this point I have worked with Cisco TAC to verify QOS is properly working along with checking everything else between point A and point B.  I again try to plead with him that is not the network causing the slowness. This time he decides to get rid of the T1’s and orders 10mbps fiber for the location. Surprise, surprise, the issue is still hanging around the application.  At this point upper management is finally starting to realize it is not a network issue and brings in an application performance monitor to pinpoint where the issue is.  Turns out the issue was the application’s database being too bloated and improperly configured.

The moral of the story is that… it is not the network.

This is a paste of a tale I told last year, but I think it fits quite well here.

Some years ago, I worked as the sole admin/IT professional for an extremely rural school district in the northern part of my state. As a matter of fact, the entire district was one school.

We had a couple of rolling wireless carts with laptops in them that circulated among the classrooms - still a pretty common scenario.

The carts also had standalone access points - these things were ancient in 2008. For reference, these were 802.11b-only, so best case is a screaming 11mbps.

One day, the science teacher had a guest in to talk to the class about some related topic. This gentleman was in his early sixties.

Mid-class, I get a call that the wireless for a bunch of the laptops seems to be not working.

I head over and take a look - sure enough, a few of the laptops will get a signal and then mysteriously drop off. I carried one over to a workbench on the other side of the room to have a closer look while the class continued, and would you look at that - the signal came back! Okay, weird. I bring it back over to the cart (surely the signal won't be WORSE closer to the access point, right?) and - no signal! I'm puzzled, and start visually inspecting around the cart, make sure the AP's plugged in, etc. All looks good.

Then I notice that the guest speaker is standing pretty close to the cart, and occasionally is right next to it. He also occasionally grimaces mildly and briefly, but continues his talk for a bit. When the talk adjourns, the science teacher mentions the grimaces and the speaker relates that he keeps 'hearing some really weird noises in his hearing aid'.

A slow light starts to dawn over the debris-strewn hills of my mind....I ask, innocently enough, if the gentleman's hearing aid uses the 2.4ghz band. He replied in the affirmative.

I bring the laptop I had been testing with to the back of the room - signal comes back. I walk toward the guest speaker, and experience a dropoff at about ten feet away.

This was the time that Bill's hearing aid broke the wireless.

Back in my collage days (1990s) we would spent a good deal of time in the Computer Labs to work on various projects. The labs were also used for class demonstrations and for personal use as well. The bad part of using the Lab was shared bandwidth. Network resources were able to see which terminals were using the most bandwidth but that only gave a PC Name like "LAB1-VT220-034".

Finding the offending machine was always a difficult task as none of the labs were well labeled or laid out. When this happened during a class "lab" I did the following after finding out the offending PC name and informing the lab instructor that the offender might be coming to see them in a moment:

>NETSEND LAB1-VT220-034 Warning - a PEBKAC error has been detected. Please inform your instructor of this error right away.

A moment later a slightly flustered user approached the instructor for the class about an error on their screen...I guess I got lucky in that the user did not know any better, but then again, sometimes the most elegant solutions come as a result of simple luck.

One day I came into work, opened up NPM, and found multiple new servers with print services available on my network.  On a subnet I didn't use internally.  What the . . .?

On a hunch I tried pinging their .1 address and found a reply.  "So who's on my network?" I wondered.  I opened a session to .1 to see if might be a router.

"Hey, I recognize that prompt" says I.  "It looks like a something I'm trained on.  And no warning about 'unauthorized access forbidden.'  I wonder . . ."

You guessed it--the default user name was in place, along with the default password.  Ever been root on someone else's router?

Without an intuitive network name or an snmp-location entry, who do I contact to fix this?  Let's see who the neighbors are . . .

Uh oh.  City government.  Police department.  Finance.

Who's the long-distance neighbors?  State Government--not good!

I did NOT want to show up on their radar as a black hat, when in fact I was a white hat!

Having learned enough, I logged out and called that city's IT department--only to find there was no one there responsible for their routers.  They contract network services out to a private company.  And it was the same one who was doing my company's WAN services!  Now I know how they got into my network, without having a clue about what was happening.

OK.  I informed the City IT folks what I'd found.  Their network had no security, default user names and passwords were being used on their routers, and I could see their departments and their access into the State government networks.  They still had no clue this was bad. . . . and it went right over their heads.

So I hung up on them and called up my WAN service provider and asked them why that City network was spanning into mine.  Why the City's network equipment was using default names & passwords, and had no security.

("mumble-mumble . . .  I'll get back to you shortly")  and they hung up on me.

NMP kept watch and pretty soon those multiple City servers were no longer showing up in my Novell world.  Then the oddball subnet disappeared, and their routers stopped being pingable.

Later I spoke informally with one of the Network Engineers for that provider, and he admitted "One of the guys spanned the City's VLAN into one of your trunked ports.  You learned their routes.  You have security enabled and your ACL's prevented them from seeing you.  We dropped the ball for their security, both in VLAN port spanning and credentials not being changed.  Thanks for letting us know--and for not telling them!"

Solarwinds products made me immediately aware of unauthorized changes to my network that day.  And it saved the day for that City's network, and for the WAN Service Provider who'd mistakenly connected two networks together.

Hi All,

I want to make a view of Node List (Resources: All Nodes) in Orion NPM, but only show down nodes that already down for 1 hour. So can anyone please help to build the filtering SQL?

The only SQL I know is to filter down node "Status=2", but I still got no clue to filter the SQL for condition this 'more than 1 hour'

Really appreciate for your help. Thanks a lot

Which method is more secure to monitor a Firewall device?

When creating Ticket Reports, I would like to have the ability to report on clients, not just creators.

If a user calls into the HelpDesk and a Tech opens a ticket for them, the tech is now the creator, not the user. When creating reports, I would like to see client data.

I just reached a goal I set for my self - over 100,00 thwack points! So I ponder...why are you collecting them?

Let us know what you thought about the Alert Central Office Hours chat.

Tell us what's up with using Alert Central in production. Are you there yet? What's stopping you? Missing features, that super annoying alerts bug that we're fixing, something else?

Do you use any other SolarWinds products, or is Alert Central your first?

The report is of interface utilization, and the sorting need to be done on following conditions.

1. Sort the report w.r.t Monday-Friday(Sat and Sun excluded)

2. Sort the report w.r.t business hours i.e 9am -5pm

3. Sort the result on monthly basis.

All need to be in same report, which includes only business days and business hours and on monthly basis.

I really need help with this. Will be much obliged to get the help.

Below is a SQL query that i received from someone, but i do not know how it works or what will be the result because the query is generating error.

declare @periodbegin datetime

set @periodbegin='17-05-2014'

declare @periodEnd datetime

set @periodEnd='18-05-2014'

SELECT

    StartTime.EventTime,

    Nodes.Caption,

    StartTime.Message,

    DATEDIFF(Mi, StartTime.EventTime,

    (SELECT TOP 1

        EventTime

        FROM Events AS Endtime

        WHERE EndTime.EventTime > StartTime.EventTime AND EndTime.EventType = 5

            AND EndTime.NetObjectType = 'N'

            AND EndTime.NetworkNode = StartTime.NetworkNode

        ORDER BY EndTime.EventTime))  AS OutageDurationInMinutes,

    DATEDIFF(Mi, StartTime.EventTime,

    (SELECT TOP 1

        EventTime

        FROM Events AS Endtime

        WHERE EndTime.EventTime > StartTime.EventTime AND EndTime.EventType = 14

            AND EndTime.NetObjectType = 'N'

            AND EndTime.NetworkNode = StartTime.NetworkNode

        ORDER BY EndTime.EventTime))  AS OutageDurationInMinutesByPower

FROM Events StartTime INNER JOIN Nodes ON StartTime.NetworkNode = Nodes.NodeID

WHERE (StartTime.EventType = 1) AND (StartTime.NetObjectType = 'N') AND

eventtime between ('17/07/2014' and '18/07/2014')

ORDER BY Nodes.Caption ASC

Best Regards.

When I head out to conventions, especially the bigger ones like Cisco Live, I always expect to find some darling technology that has captured imaginations and become the newest entry in every booth denizen's buzzword bingo word list. And most of the time, my expectation is grounded in experience. From SDN to IoT, and on through cloud, container, and BaaS Blah Blah as a Service (BaaS), each trend is heralded with great fanfare, touted with much gusto, and explained with significant confusion or equivocation.

Not this year.

Chalk it up to the influence of Berlin's tasty beer and solid work ethic if you want, but this year the crowd was clearly interested in "the work of the work," as I like to call it, or "less hat, more cattle," as my friends in Austin might phrase it.

Don't get me wrong. The sessions were engaging as ever. The vendor floor was packed. The attendees came early and stayed each day to the end. The DevNet area was bigger than ever before. It was, by every measure, a great conference.

More about DevNet: While there were a lot of younger faces, there was no shortage of folks who clearly had put their years in. Patrick was the first to notice it, and it's worth highlighting. Folks with depth skills in a technical area were taking time to begin training on the "new thing," a set of skills that are up-and-coming, which do not match, in any way, the techniques they use right now, which may not even bear a resemblance to their current job. But they were there, session after session, soaking it in and enjoying it.

But as I commented to patrick.hubbard and ding, I hadn't yet found "it." And they both pointed out that sometimes the "it" is simply thousands of people spending time and money to come together and share knowledge, build connections, and enjoy the company of others who know what they know and do what they do.

Meanwhile, a steady stream of visitors came to the SolarWinds booth asking detailed questions and waiting for answers. Sometimes they had several questions. Often, they wanted to see demos on more than one product. They ooh'ed and ah'ed over our new showstoppers like NetPath and PerfStack (more on those in a minute), but stuck around to dig into IPAM, VNQM, LEM, and the rest.

After speaking to someone for a few minutes, visitors were less apt to say, "Can I have my T-shirt now?" and more likely to say, "I would also like to see how you can do ______." For a company that staffs its trade show booths with an "engineers-only" sensibility, it was deeply rewarding.

But there was no "trendy" thing people came asking about. There simply was no buzz at this show.

Unless - and I'm just throwing it out there - it was US.

You see, about a month before Cisco Live, SolarWinds was identified as the global market share leader for network management software (read about it here: http://www.solarwinds.com/company/press-releases/solarwinds-recognized-as-market-leader-in-network-management-software). Now that's a pretty big deal for us back at the office, but would it matter to in-the-trenches IT pros?

It mattered.

They came to the booth asking about it. To be honest, it was a little weird. Granted, a kind of weird I could get used to, but still weird.

So it turns out that Cisco Live didn't feature a buzz-worthy technology, but instead we found out that we got to be the belle of the ball.

PostScript: Next year, Cisco Live will be in Barcelona, Spain. Espero ver tu alli y hablar con tu en ingles y español.

We use, and re-sell Mitel VOIP solutions. It was be fantastic to be able to sell an all encumbered solution including monitoring to our clients.

My company uses Cyberark and when talking with the Solarwinds Engineers, they seem unsure of how to pull credentials from Cyberark and create them in Solarwinds. This is sort of a deal breaker for us and I REALLY want to switch off of our current solution to Solarwinds.

HI,

Anyone using NCM to manage ForeScout, Gigamon, DataPower or RiverBed configurations?   We currently do not own NCM and are wondering if anyone has had success using NCM to manage these devices.....one of our key decision points on whether to purchase NCM.......

Appreciate any feedback/thoughts on these....

we are implementing SAM 6.4 and i would like to know what levek of access does the solar winds SAM require to monitor and run reports under.