Friday, May 02, 2014

Wireshark, GeoIP and Checking Up on Mobile/Home Carriers

As enterprises move an ever-growing list of services into the mobile space, it becomes essential to understand the limitations of the mobile network infrastructure.  No longer can we perform true end-to-end capture or analysis of network data; what was the "last mile" is now an indeterminate path through any number of relatively impenetrable mobile networks.  In this respect, troubleshooting issues involving mobile devices can be quite the challenge. At the same time, we're dealing with an increasing number of telecommuters, those "work from home" people who are at the mercy of their ISP.  What, then, is the enterprise network analyst to do?

The answer (or, at least, a good start toward an answer) lies in geolocation - the association of IP address spaces with their geographic and/or corporate assignments. Geolocation can be been integrated with DNS (or, at least, BIND implementations of DNS), the Apache web server, and any number of other applications, including (as of version 1.1.2) our favorite network tool - Wireshark. The marraige of Wireshark's analysis and GeoIP's provider identification produces some powerful analysis capabilities.

You can download free GeoLite versions of current GeoIP databases from MaxMind.  MaxMind provides free GeoLite databases for IPv4 and IPv6 city, country and autonomous system numbers (ASNs); you'll want to download the binary versions, not the CSV editions.  The MaxMind databases are updated on a monthly basis; if you like the results of this exercise, you'll need to set up a process to handle monthly updates.

Now it's time to make Wireshark GeoIP-aware:

1) Once you've downloaded the GeoIP databases, unzip them to a permanent home. On my Linux systems, I created the /usr/local/geoip directory for this purpose; on Windows systems, I use a \geoip subdirectory under the Wireshark installation directory.  The databases can be (and should be) read-only; you won't be adding any data. Now, we're ready to pull them into Wireshark.

2) Open Edit->Preferences in Wireshark, select Name Resolution, and click the "Edit" button next to GeoIP database directories; click New in the resulting dialog and add the directory you created in step 1. Using my Linux example above, you should have something like this (click to enlarge):
3) Close Wireshark and reopen.  You're ready to go!

So, what exactly does this give you? Well, to start with, you'll find that Wireshark's Statistics->Endpoints includes sortable columns for City, Country and AS Number, like so (click to enlarge):
You'll also find GeoIP information in the Details pane of the packet view, under Internet Protocol:
Finally, you can now use GeoIP information in your Wireshark display filters. For instance, I'll take the ASN definition from the previous example (British Sky Broadcasting) and use it in a display filter to show me ALL traffic from that provider:
ip.geoip.src_asnum == "AS5607 British Sky Broadcasting Limited"
Other GeoIP display filters allow you to select/view traffic based on country (e.g. ip.geoip.country=="Egypt"), city (e.g. ip.geoip.src_city == "Birmingham, AL") or even longitude or latitude (e.g. ip.geoip.src_lat == 33.520699).

From here, you can isolate, analyze and/or export data for specific providers, whether they serve mobile or home users; you could even develop "country profiles" if you're serving an international clientele.  While the GeoIP data isn't perfect, it's more than adequate to help you create a profile of your mobile userbase.

Have fun!

Thursday, May 01, 2014

Network Troubleshooting - Sometimes It's What You DON'T See...

I spend a healthy chunk of my typical work day analyzing network packet captures.  My primary tool is Wireshark, which humbly presents itself as "The World's Most Popular Network Protocol Analyzer."  (Seriously - if you aren't using Wireshark, go download it NOW.)  Protocol analyzers are great for identifying typical "red flags" in packet data, but they're all limited to what the raw data might indicate; customer network environments are so broad (and so varied) that the network engineer--especially one "on the outside looking in" with only a small data set--relies heavily on experience and intuition.

One recent case was presented as "many failed connections," and a 6-minute packet capture soon landed in my lap.  Now, every Wireshark user has their own approach; I usually take advantage of Wireshark's display filters to get a general "feel" for the incidence of Layer 3/4 problems. With a typical capture file, I'll start with tcp.analysis.flags,which simply tells Wireshark, "hey, show me what YOU think are TCP problems." Now, as I said, none of these tools are perfect, so take these results with a grain of salt; they're only as good as are the underlying data, and it's very easy to collect inaccurate or incomplete data. After taking a look at the results of this display filter, I noticed what seemed an high number of TCP retransmissions, so I decided to see exactly which packets were being retransmitted with a different display filter, tcp.analysis.retransmission, which will show me only those packets Wireshark believes to be TCP retransmissions. The resulting numbers were somewhat high, but I've seen worse. Now, the complaint was very specific that new connections were failing; no mention was made of existing connections being interrupted/terminated; so, I went to Wireshark's Statistics->Conversations dialog and sorted on the "Packets" column to look for very short conversations and found HUNDREDS of conversations that only lasted for a few packets, like these:
Well, now, wait just a minute - the TCP handshake requires 3 packets (SYN, SYN/ACK, ACK) to establish a conversation, and I'm seeing hundreds of conversations that are only exchanging 3 to 6 packets. After checking a few suspect conversations, I found a pattern, namely this:

So, the remote endpoint starts a conversation with a SYN packet and the local endpoint responds immediately, but we see the remote endpoint retransmitting its SYN packet within 10ms. The local endpoint retransmits its SYN/ACK, but neither the original nor the retransmitted SYN/ACK seem to reach the remote endpoint, and the conversation attempt is ultimately terminated with a TCP reset (RST) packet. Back I go to Wireshark's display, this time to ask about a very specific type of TCP retransmission:
tcp.analysis.retransmission && tcp.flags.syn==1 && !tcp.flags.ack==1
With this display filter, I'm asking Wireshark to show me all retransmitted SYN packets; the "!tcp.flags.ack==1" eliminates SYN/ACK packets from the display. The results were startling; within a 6-minute period, more than 110 endpoints had retransmitted more than 170 SYN packets...and all of them had failed to complete the TCP handshake.

Well, if conditions are this bad to START conversations, then there must be thousands of cases in which existing connections die before completing successfully, right?  Let's go back to Wireshark's Statistics->Conversations dialog and sort on Duration to look at long-lived conversations:
Hmm...I have hundreds of conversations that last longer than 2 minutes...but I can't find one that suffers from retransmissions sufficient to terminate the conversation.

If I were looking at a general network congestion issue on the local network, I'd expect conversations to suffer equally--packets are packets, right?--but this is something different. That seeming conflict in the data prompted what proved to be the key question:
If I'm seeing HUNDREDS of new conversations fail the TCP handshake due to excessive retransmissions, why DON'T I see established conversations suffering excessive retransmissions as well?
Well, after few moments' thought, it occurred to me that the only network devices that usually make specific distinctions between new and existing connections are those involved in network security. A brief conversation with the customer revealed that an intrusion protection system (IPS) was in place and "inspecting" conversations. When we conducted a test that bypassed the IPS, the incidence of failed TCP handshakes decreased by roughly 98%; our troubleshooting attention is now properly directed.

So, the moral of this story: Pay attention to the data, but pay equal attention to what isn't there.

Wednesday, March 05, 2014

SxSW = Tons of Free Music!

SXSW Music: South By Southwest 2014 | Lineup | Rumors | Tickets | Film Festival | Dates | Mobile App | Video | Austin | TexasAustin, Texas is cranking up South by SouthWest 2014, and that means music - LOTS of music. SxSW gets most of its mainstream press for its technical and creative content these days, but the festival's roots are musical; hundreds of bands will be playing in dozens of venues. (Can you tell that I REALLY wish I were attending?) As you might expect from a festival that features so much indie music, there are more free samplers and promotional releases than anyone could possibly want to download...unless, of course, they're a total music geek. So, since I'm already downloading these (grin), I thought I'd share the links with you. In the interest of brevity, I won't go into lengthy descriptions of each collection; the only thing I'll say is that I inevitably find REALLY good music, across a range of genres, in every year's samplers. Having said that, let's get to the list: 

NPR's Austin 100 - A massive (839 MB) collection of NPR's picks. 
Arts & Crafts Records - This Canadian label will have a healthy presence in Austin. 
Bar/None Records - Always an interesting selection.  
SxSWBaby! - Another large (80 tracks, 314MB) collection from the thoroughly UNofficial SxSW blog. 
Rubberneck/Burger City - Truth be told, this one is brand-new to me; I'm downloading the tunes as I write this...  
Dine Alone Records 
OK! Records 

Some of these are straight downloads, while others require your email address. Enjoy!

Wednesday, January 15, 2014

#SpeakerU - Speaker Coaching: It's Worth The Time

OK, so, you've been selected to speak at a conference.  You've spent hours building your slides, you've gone through your outline a dozen times, and you've even jotted down particular words (or turns of phrase) you want to use.  There's only one problem - you've never really given this presentation to an audience.

Enter the speaker coach. 

Now, everyone seems to have their own idea of what a speaker coach should be; as far as I'm concerned, a good technical speaker coach should be able to do two things:
  • Critique the mechanics of your presentation style, and
  • Evaluate the technical content of the presentation itself.
The former category can be tricky for technical folks.  It seems that we're used to people speaking fast, using run-on sentences, and throwing jargon about with reckless abandon.  Most of us have to take a breath and slow things down a bit when speaking to an audience that isn't as familiar with the subject matter as we might be.

The latter focus--the technical content--should be evaluated in the context of the intended audience.  If you're targeting your presentation to beginners, the question is, "Did they get from [nothing] to [basic understanding]?"; presentations to folks with intermediate skills should result in the listener "getting to the next level," and speaking to advanced audiences...well, at that point it's usually about bringing something new to the table, since they (by definition) already know the beginning and intermediate stuff, right? So, you want to find someone who fits the "typical attendee" model, NOT someone who already knows the material.

Long story short, folks - find a person (or persons) who can do these two things for you.  Take them through your presentation.  Don't stop, don't pay any attention to the fact that there are only one or two people in the room, and don't hold anything back: give the presentation as you expect to deliver it on stage.  If you can get a projector, do so; make this as realistic a "dry run" as possible.  After you're done, sit down with them and ask two questions:

  1. "Could you hear and understand my words and how I spoke?"
  2. "What can you tell me about [subject of presentation]?"
Their answers will tell you what (if anything) you need to change.

I can't say this strongly enough: DO THIS.  DO THIS SEVERAL TIMES.  Every runthrough you can make before hitting the stage will improve your performance.

Now, go practice!

Wednesday, September 18, 2013

#SpeakerU - Building Your Deck, or "You Want Slides With That?"

Previously, on #SpeakerU...
So, you sucked up your courage, submitted a proposal to a meeting or conference, and then--O happy day!--you received that wonderful "your abstract has been accepted" message.

Now you must...MAKE SLIDES.  You may need to make MANY SLIDES.

Here's where you really have to start thinking about the mechanics of your presentation: how much time you'll have, what points you MUST cover, your own comfort level with the information, how much time to spend on each piece of information...all of this goes into the slides you're about to create.

At this stage of the game, one's slides often take on the 'personality' of the speaker.  This is, I think, as it should be.  The goal of #SpeakerU is not that everyone who reads these articles churns out presentations of identical style--that would result in really boring conferences, if you ask me--but rather that your slides balance your speaking style with the material you need to cover.  Remember, those folks are coming to hear YOU; if they only wanted the raw information, they could have stayed at home and read a book.  Find that comfort level that lets you feature the content.

I'll also recommend a visit to Slideshare, where folks from around the world have posted presentations on a wide variety of topics.  There's nothing wrong with looking at the different formats and styles on display to find your way, but resist the temptation to copy slides wholesale.  (You WILL be caught eventually; remember that the attendees can check out Slideshare, too...)

Instead of going through the mechanics of building slides or talking about point sizes and the like, I'll leave you with some general guidelines I use when laying out the slides for a presentation.  Feel free to use or adapt them as you see fit; your deck is about your style and your information, not mine.

So, then, in no particular order:
  • Assume that your audience can read.  Reading your slides is THE fastest way to lose your audience.
  • Make a noticeable transition, either vocally or by changing slides, every 90 seconds or so - or you'll start losing people.
  • Resist the urge to use copyrighted material of any sort.  Imitiation is the sincerest form of flattery, but it's also the fastest path to a conversation with lawyers.
  • If you can't cover a one-topic slide in less than 2 minutes' time, split the information across multiple slides.
  • Only use effects/animations when they deliver a necessary accent or transition to the content - in other words, forget the eye candy.
  • Don't be afraid to use a (well-designed) graphic instead of words, but remember to give enough written information to spark the listener's memory when they read your slides a month or two later.
  • Don't "get fancy" with language. You'll be seen as a show-off, especially where words/phrases from other languages are concerned.
  • Remember that you will be "talking AROUND your slides," adding perspective/depth to what your audience is reading.
  • Don't let your slides "skip around" - don't take your audience to Point C if you need to talk them through Points A and B first.
  • If you use graphics, remember that they need to be clear to the folks in the last row - avoid clutter.
  • Be VERY careful with humor, especially when delivering to an international audience.
  • Don't create eye charts; use a reasonable font/size for the projection system that will be in use, and stick with it.
Once I've made a "rough draft," I usually put it to two tests; I hand it to someone who knows the topic well ("Did I miss anything?"), and then hand it to someone who doesn't know the material at all and ask, "Can you follow the general flow?"  Remember that your audience will probably cover the spectrum from 'newbie' to 'expert;' you want to inform the former without losing the latter.

Next up - putting it all together...

As always, feel free to add your own experiences, tips, or pasta recipes in the comments...

Wednesday, July 10, 2013

#SpeakerU - Writing an Abstract, or "PLEASE PICK ME!"

It's time to get back to #SpeakerU, our ongoing discussion of how you can "get up on stage" and become a technical presenter/speaker at conferences like IamLUG (now ICONUS), ICONUK, and IBM Connect.  (If you want to play catchup, my previous #SpeakerU articles are here and here. Several other folks have written articles, as well; a quick search for "#SpeakerU" should bring their work to your attention.)

Most conferences do NOT expect you to have a full deck of slides (or a complete paper, or a finished poster) prepared in advance; instead, they'll ask you to submit an abstract - a brief explanation of your topic and material.  Now, whether you realize it or not, many conferences make the abstracts of accepted sessions public well before the conference meets, so that prospective attendees can begin planning their schedules.  Thus, your abstract has to do two things:
  • Convince the folks hosting the conference that you know your stuff, AND
  • Convince attendees to attend your session.
Your task is complicated by the fact that abstracts are usually limited in size; you might have as few as 500 characters with which to "make your pitch."  (Remember in the last article, when I said that a big part of picking a topic was fitting it to the time available?  Well, here's another reason to choose wisely; you have to be able to explain it succinctly!)  So, how are we going to accomplish so much in so few words?

I tend to use abstracts to do one thing - ask/answer attendees' questions.  If you've identified where folks are having problems AND can deliver solutions, that's going to be a big win with the conference organizers; if you hit upon a common problem (or a Frequently Asked Question), you may hear a buzz about your presentation well before the conference and (eventually) face a packed room.

Instead of creating spurious examples at this point, I'm going to point you to Russell Maher's recent article on the subject.  He gives the best possible example of the abstract process - his own abstracts. (As you'll see, I collaborated with him.)  The key point is take your topic and put it in terms of the attendees' environments, their questions and your answers; remember that they may not even realize that they need to visit your session until they read your abstract.   It isn't what you can teach - it's what they can learn if they attend.  Russell provides a before-and-after view of two session abstracts, which I think illustrates the process pretty well.

For all this talk about 'selling' and 'pitching' your session/presentation, it's important NOT to oversell.  Remember, you can't claim to make attendees experts in a one-hour session, nor can you ever answer "all the questions."  Be reasonable and straightforward about what you can deliver; as I've often said on stage, sometimes the best outcome of a presentation is NOT that everyone finds their questions answered, but rather that they learn the right questions to ask!

A good abstract requires effort; it isn't something you dash off in 90 seconds.  If you can find old conference programs, take a look at successful abstracts from past events.  Ask your friends to review your first effort, ask others to rewrite them, and don't be afraid to take it through several iterations.  The important thing is NOT to give up - keep trying!

Next up: You've Been Chosen - And Must Now Create Slides...

Tuesday, July 09, 2013

Puzzling Through Music? Here's My Chordspeller...


If you've been paying attention to this blog (or if you've ever met me), you know that I'm a complete music nerd.  Well, that extends to music theory; I'm one of those folks who fakes playing piano by cheating from the guitar notation, and I'm one of those singers who wants to know whether I'm singing the root, third, fifth or seventh of that big chord at the key change.  Well, to do that stuff one needs a fairly comprehensive chord library, whether it be a mental storehouse or a cheat sheet.  I'm not that good - I need a cheat sheet.

Long ago, when I first started dissecting barbershop music (and, yes, baritones DO have the strangest part!), I had a great 'chord spelling' chart; unfortunately, it was lost some years ago and I've never been able to find a good replacement.  My daughter recently ran across some 'fake sheets' for current pop songs, which have naught but lyrics and guitar chords; she started asking me "What's a Cdim7?  How do you play an Fm9b5?"  I thought of my old chord speller, and decided to create my own.

Simply put, it's a one-pager that includes:

  • Nomenclature (e.g. dominant 7th, minor 6th, etc.),
  • Symbol notation (e.g. Cm7, C+, etc.), 
  • Scale note formula (e.g. C minor triad = "1 b3 5"), and
  • Chord listings (e.g. "Gbm9 = Gb A Db E Ab")
for 204 chords, plus extra notation & scale note formulae for another 15 chord families, for a total library of 429 chords - on a single 8.5"x11" page.

You can grab a PDF copy of Chordspeller from my Slideshare library.