Speed Scripting - EXIF, JSON, and OpenBadges

As OpenBadges have become more commonplace, I find that I haven't earned them all in a single identity; some were awarded through my work email address, others via my primary personal email address, and still others to yet another personal email address.  This led to a bit of confusion in a recent course of study, when I accidentally completed the prerequisite work under multiple email addresses.  To resolve that problem (and ensure that I don't do that in the future), I needed a tool to display the email address to which each of my badges had been awarded.

Now, there are a few complicating factors:

1. This information is stored (in the badge PNG image) as metadata, namely a JSON array.  We don't yet have any standard/common Linux command-line tool for parsing JSON, and I like to write scripts with as few dependencies as possible.
2. The OpenBadges specification has been around long enough to go through a few versions, and the location of the information has changed from version to version; that means I can't do a straightforward "it's always in field 27" text-based extraction.
3. For verification purposes, the recipient's email address is NOT stored as plaintext; rather, a SHA256 hash of the address is embedded.  So, I'll need to know the SHA256 hashes for each of my email addresses used in earning badges; I used this free SHA256 Hash Generator to grab the necessary hashes. 

First, I installed exiftool (which is available for most Linux distributions - check your package manager) to extract the PNG metadata.  The OpenBadges JSON array is returned as a single string element, like so:

Openbadges                      : {"@context":"https://w3id.org/openbadges/v1","type":"Assertion","id":"https://www.youracclaim.com/api/v1/obi/badge_assertions/c1396602-e7c9-4213-a769-7d2735fb0f28","uid":"c1396602-e7c9-4213-a769-7d2735fb0f28","recipient":{"type":"email","identity":"sha256$cb6c6af3883a452da960b6cb88630b221888e7afce0ff612a483351664bb7dd0","hashed":true},"image":"https://acclaim-production-app.s3.amazonaws.com/images/84ac9eff-b8a2-4683-846b-f59887a73801/Python%2B101%2BData%2BScience.png","evidence":"https://www.youracclaim.com/badges/c1396602-e7c9-4213-a769-7d2735fb0f28","issuedOn":"2017-12-12T23:57:06.000Z","badge":"https://www.youracclaim.com/api/v1/obi/badge_classes/91e77961-2bcc-4b3a-9ff8-9333921bb2c4","verify":{"type":"hosted","url":"https://www.youracclaim.com/api/v1/obi/badge_assertions/c1396602-e7c9-4213-a769-7d2735fb0f28"}}

The only piece of this that interests us (in this case) is this:

sha256$cb6c6af3883a452da960b6cb88630b221888e7afce0ff612a483351664bb7dd0

Now, this series is entitled "Speed Scripting", so I'm not about to write a full JSON parser.  *chuckle*  Given that this is basically a string of text output, I'm going to use grep to pull the Openbadges string from exiftool's output and employ awk to iterate through the fields and print only the field containing "sha256", using the double quote as my field delimiter.  (I'm using " as my field delimiter so that I don't have to go through an extra step to strip the quotes from the output string.)  The resulting string will be stored in the $hashaddr variable:

BADGEDIR=/home/wmorgan/badges
for file in $BADGEDIR/*.png
do
  hashaddr=`exiftool $file | grep Openbadges |
  awk -F'\"' '
     {for (i=1; i<=NF; i++) {
        if ($i ~/sha256/) print $i;
     }}'`
done

(Note that I had to use a backslash to escape the doublequote character in the -F option to awk.)

That just outputs all the SHA256 hashes...but that's only the halfway point; I need to map those hashes to their corresponding email addresses.  So, I'll add my "known hashes" that I generated earlier (note that we must prepend "sha256$" to each hash string and escape the $ character):

MYHASHEDADDR1="sha256\$caceb56680cabe892cdc5b903b2cbaf9ec3462699dff9094cc25ad27ef824aaa"
MYHASHEDADDR2="sha256\$cb6c6af3873a452da960b6cb88630b221845e7afce0ff612a483351664bb9bd0"
MYHASHEDADDR3="sha256\$2a196380936706eac9c60c638be63409be9eb2728ed0302188ce2d899ed22afb"

and use the shell's case construct to handle the comparison and output:

case $hashaddr in
     $MYHASHEDADDR1)
echo wesbo@email.address.edu
        ;;
     $MYHASHEDADDR2)
        echo wessinator@second.email.org
        ;;
     $MYHASHEDADDR3)
        echo bigwes@yetanother.emailaddr.com
        ;;
     *)
        echo Unknown
        ;;
  esac

Finally, I need to print the name of the badge with its corresponding email address, so I'll extract that from the filename with basename and print it at the top of my do loop:

echo -n `basename $file .png`": "

So, here's our finished product:

#!/bin/sh
#
# dispbadgeaddr - show email address to which OpenBadge was issued
# requires: exiftool
# prerequisite: MYHASHEDxxxxx contains SHA-256 hash of email address

MYHASHEDADDR1="sha256\$caceb56680cabe892cdc5b903b2cbaf9ec3462699dff9094cc25ad27ef824aaa"
MYHASHEDADDR2="sha256\$cb6c6af3873a452da960b6cb88630b221845e7afce0ff612a483351664bb9bd0"
MYHASHEDADDR3="sha256\$2a196380936706eac9c60c638be63409be9eb2728ed0302188ce2d899ed22afb"
BADGEDIR=/home/wmorgan/badges

for file in $BADGEDIR/*.png
do
  hashaddr=`exiftool $file | grep Openbadges |
  awk -F'\"' '
     {for (i=1; i<=NF; i++) {
        if ($i ~/sha256/) print $i;
     }}'`
done
  case $hashaddr in
     $MYHASHEDADDR1)
echo wesbo@email.address.edu
        ;;
     $MYHASHEDADDR2)
        echo wessinator@second.email.org
        ;;
     $MYHASHEDADDR3)
        echo bigwes@yetanother.emailaddr.com
        ;;
     *)
        echo Unknown
        ;;
  esac
done

and here's a bit of its output:

ibm-clm-for-safe-level-1.1: wesbo@email.address.edu
ibm-cloud-essentials: wessinator@second.email.org
ibm-cloud-kubernetes-service: wessinator@second.email.org

If I wanted to polish this a bit, I'd rewrite the core exiftool/grep/awk logic as a standalone shell function, then add the capability to specify particular PNG files on the command line or use -a to iterate through all the badges.  For now, though, I can just use the script as is and use grep to extract info as needed...

$ ./dispbadgeaddr | grep blockchain
ibm-blockchain-consulting: wessinator@second.email.org
ibm-blockchain-essentials: wessinator@second.email.org
ibm-blockchain-scale-v1: wessinator@second.email.org
interskill-blockchain-foundations: wessinator@second.email.org

Total time - 10-15 minutes...and that's what speed scripting is all about.

Speed Scripting - Making an Animated GIF with ImageMagick

Those of you who follow me on Twitter probably know that I'm something of a geek-of-all-trades; I love reading and learning.  When my employer, IBM, entered the world of Open Badges for employee credentials, I jumped in with both feet.  We're encouraged to display our badges in email signatures, online, et cetera...but what is one to do when one has earned 50+ badges in a wide range of areas/disciplines?  Now, obviously I can't put that many badges in an email signature, and a static display can only do so much...so the thought occurred to me, "I should be able to turn these into an animated image!"  Time for some shell scripting...

Let's consider the source material:

• Our badging provider provides badges as 352x352 PNG images, so I don't have to worry about resizing individual images; I can treat them all the same way and resize the end product as needed.
• I'm not worried about the PNG metadata in this particular case, so converting to another format isn't a big deal.
• I could go with an animated PNG (APNG), but (a) not all browsers display APNGs properly, and (b) animating PNGs can result in images significantly larger than comparable animated GIFs

At this point, I'm ready to start playing with ImageMagick, a wonderfully complete (and open source!) image manipulation package.  ImageMagick handles 200+ image formats, and is available as both a command-line toolbox and libraries/interfaces for a wide range of programming languages.

The actual construction of the animated GIF with ImageMagick is straightforward - it's a one-liner:

$ convert *.png -frame 5 -dispose Background -set delay 150  allmybadges.gif

The options put a frame around each image, dispose of the previous image before displaying the next one, and set the delay (in hundredths of seconds) before adding them to the end product, allmybadges.gif.

Wait, though...oh, I don't like that.  Since many badges have similar names (e.g. task-level-1.png and task-level-2.png, ibm-iot-whatever1.png, ibm-iot-whatever2.png, etc.), my use of a wildcard puts them in alphabetical order...which is kind of boring when 2-3 consecutive badges share color schemes or other design elements.  So, what I really want to do is randomize the badges before animating them.  Well, bash provides $RANDOM as an environment variable, yielding pseudorandom integers between 0 and 32767; that should be sufficient for me to play with filenames.

I wound up with this script:

#!/bin/bash
#
# badgeanim - create animated GIF of all PNG badges
#
WORKDIR=/home/badges/animate
BADGEDIR=/home/badges

cd $WORKDIR                          # move into working directory
rm $WORKDIR/*.png                    # blow away any old PNG files

for file in $BADGEDIR/*.png          # grab copies of current PNGs
do
cp $file ./$RANDOM`basename "$file"` # prepend $RANDOM to filename
done

convert *.png -frame 5 -fill snow2 -background snow2 -dispose Background -set delay 150 -set dispose Background allmybadges.gif                                         # ImageMagick builds the GIF

rm $WORKDIR/*.png                    # blow away leftover PNG files
exit

The results can be seen at the top of this article and in the "My OpenBadges" widget on the right margin of the page.  (Clicking on the animated GIF to the right takes the viewer to a static display of all badges.)  I could resize the end product for use elsewhere, like so:

$ convert allmybadges.gif -resize 200x200 allmybadges-200x200.gif

Check out ImageMagick - it's definitely worth your time if you do anything at all with images.

First Steps in node.js - Fortune Cookies!

I've been doing a lot of learning around cloud architectures, microservices, and the like, so I thought it was time to start learning JavaScript and node.js.  I started working with IBM's SDK for node.js, but soon learned that IBM has deprecated its proprietary SDK in favor of the community SDKs. I installed the latest/greatest community SDK (v10.8.0) on my Ubuntu Linux system.

Some months ago, I wrote a Twitter bot that delivers a random fortune-cookie quote/saying every 6 hours. (It's @CollectedQuotes, if you'd like to follow it.)  Well, what better example could I adapt as a simple service in node.js?  Here we go, short and sweet (click images to enlarge)...

Here's the source code:

Here's the (very simple) browser page it generates:

Here's the 404 response to any URL request other than the root:

Finally, here's the console log:

It might be clunky, but it works!  If you'd like to play with this, you can download the JavaScript source and the fortune-cookie file.

Optimizing Wireshark for HTTP Analysis/Troubleshooting

I spend quite a bit of time troubleshooting various web applications, so I've done a lot of work with Wireshark's HTTP display filters. The ones I use most frequently are:

• http.response.code - In an HTTP response, the numeric response code (200, 404, 500, etc.)
• http.content_length - In an HTTP response (and PUTs and POSTs), the total payload size
• http.request.method - The type of request made by the client (e.g. GET, PUT, POST, CONNECT, etc.)
• http.request_in - For an HTTP response, the packet # of the corresponding request
• http.response_in - For an HTTP request, the packet # of the corresponding response
• http.time - elapsed time between request and response

That last filter needs a bit of explanation, because it can be computed in two different ways.  If the TCP preference "Allow subdissector to reassemble TCP streams" is enabled, http.time reflects the time between the request and the last packet of the response (i.e. the end of any data returned); if that preference is disabled, http.time reflects the time between the request and the first packet of the response (i.e. the HTTP response code).  I almost always have that TCP preference enabled, because the client/browser usually can't do anything with the response until it receives all of it!

So, one could work through a Wireshark session, using display filters like http.response.code==404, http.content_length > 4096, or http.time > 2.0 to display various packets...but, to be honest, I'd rather not do that much typing.  So, I set out to optimize Wireshark's performance and tweak its display for HTTP analysis.

The most significant performance optimization one can implement in Wireshark is to disable analysis of irrelevant protocols.  By default, Wireshark tries to dissect every protocol it can identify...and there are hundreds of protocols in its dissection engine.  Disabling irrelevant protocols will greatly enhance overall performance.  Keep in mind, though, that one needs visibility at all layers of the network stack; for most work in my environment, I need Wireshark to dissect Ethernet, IPv4, TCP, UDP, ICMP, SSL/TLS, HTTP, and a handful of other protocols.  I disabled analysis of all other protocols.

Now, to the GUI tweaking.  First, I created filter buttons for general display filters, so that I can apply them with a single click:

• http - apply http - display all packets identified as HTTP
• 1xx - apply http.response.code < 200 - display responses with informational codes
• 2xx - apply http.response.code > 199 && http.response.code < 300 - display responses with success codes
• 3xx - apply http.response.code > 299 && http.response.code < 400 - display responses with redirection codes
• 4xx - apply http.response.code > 399 && http.response.code < 500 - display responses with client error codes
• 5xx  - apply http.response.code > 499 - display responses with server error codes
• >2s - apply http.time > 2.0 - display all responses that required more than 2s to complete
• GETs - apply http.request.method=="GET" - display all GET requests
• POSTs - apply http.request.method=="POST" - display all POST requests

Then, I added columns for use in "eyeballing" HTTP traffic and doing quick sorting (you can sort on any column in Wireshark's display):

• Stream - display tcp.stream, the connection identifier generated by Wireshark as the file is read
• Req # - display http.request_number, the request's sequence number within its connection (useful for HTTP-pipelined connections)
• HTTP Req - display http.request_in for HTTP responses
• HTTP Res - display http.response_in for HTTP requests
• HTTP Time - display http.time
• HTTP RC - display http.response.code for all responses
• Payload - display http.content_length for all requests/responses with data payloads

Here's a sample of the results (click to enlarge):

(You can see the "one click" filter buttons in the top right.)

From here, I can sort on any column (like, oh, HTTP Time?), match requests and responses easily, identify "red flags" at a glance (why on earth did it take 1.5s to pull down PNG files of only ~280 KB?!), and examine how "red flags" affected subsequent requests on the same connection (that 1.5s delay affected the browser's processing of the next GET as well, since they were on the same HTTP-pipelined stream), all from a single view...and Wireshark's processing time is greatly improved, to boot!

Now, this isn't perfect.  If any problem occurs in IP/TCP/SSL-TLS/HTTP reassembly (for instance, a missing or corrupted packet), you won't get full information on affected HTTP transactions...but it's easy to right-click on the "missing information" packet and use Wireshark's Follow TCP stream command to zero in on a transaction with missing information and determine what happened.

(If you'd like to try this profile, you can download it as a ZIP file.  You'll want to unzip it in your Wireshark profiles directory; it will create a profile directory named 'HTTP'.  Within Wireshark, you can switch profiles by clicking on "Profile: Default" in the bottom-right corner.  Remember, though, that any configuration changes you make are automatically saved to the current profile; be careful!)

So, that's the Wireshark environment I'm using for HTTP analysis.  Did I miss something you consider important?  Is there a change you would make in your environment?  Let me know in the comments...

Today's Scripting - Extracting HTTP Performance Data from Wireshark with Python

I'm often asked what kind of data can be exported from Wireshark, especially when we're troubleshooting performance issues.  Most recently, someone said, "It would be great if we could suck this HTTP timing data into a spreadsheet." I'm not a big fan of spreadsheets, but I said to myself, "Hmm...can't be that difficult to do"...and sat down to write some code.

A traditional shell script with a few Linux/Unix utilities could do this (I did a LOT of awk and sed back in the day...), but I'm in the process of teaching myself Python, so I set out to do some snake charming.  I'm DEFINITELY a novice, so I'm sure that those more experienced with Python will offer improvements to my brute-force, trial-and-error code. Having said that...

The first step is to collect network packet data while also collecting the TLS session keys from the browser.  (If you haven't done that before, check out my brief video on the technique.)  For ease of use, we're going to name the packet capture files "testX.pcapng" and the matching TLS key logfile "testX-keys."

Now, we turn to Wireshark's command-line kin, tshark.  Since the timing information we need is computed by Wireshark (it isn't in the native packet data), we'll need to run a two-pass analysis in tshark.  So, our tshark command looks like this (I'll split lines for readability):

/usr/local/bin/tshark -2                    # 2 passes to pick up computed values
   -o tcp.check_checksum:FALSE              # tshark ignores packets that fail TCP checksum, so skip that check
   -o ssl.keylog_file:testX-keys            # here's our TLS session keyfile
   -Y "http.time || http.request.full_uri"  # find packets with these fields - requests have URI, responses have http.time
   -T fields                                # we want to output certain fields (specified with -e)
   -e frame.number                     # output frame number (from all packets)
   -e http.request.method              # output HTTP method (GET, POST, etc.) (from requests)
   -e http.request.full_uri            # output full URI requested (from requests)
   -e http.response_in                 # output frame # of response (from requests)
   -e http.request_in                  # output frame # of request (from responses)
   -e http.time                        # output elapsed time (from responses)
   -e http.response.code               # output HTTP response code (from responses)
   -E separator=,                      # separate output fields with commas
   -r testX.pcapng                     # read from testX.pcapng

With these options, tshark provides an output stream of mixed requests and responses that looks like this:

2324,GET,https://apps.na.collabserv.com/,2340,,, (request - note empty fields)
[...any number of intermediate lines...]
2340,,,,2324,0.059056000,302 (response - note empty fields)

(I didn't yet know it, but this format was going to come back and bite me - stay tuned...)

So, I'm going to have to match requests and responses to aggregate the needed data into a single CSV line - let's go write some Python!  In a fit of originality, I named the script httpstats.

First, basic housekeeping.  In general use, I'm assuming that the packet capture files are <name>.pcapng, and that the TLS session key logfiles are <name>-keys...so I'll expect the user to invoke my script with "httpstats name" and go from there. After importing the Python libraries I'll need, the first steps are to validate the command-line argument and let the user know where the output is going:

#!/usr/bin/python3
import sys
import os.path
import subprocess
import csv

if len(sys.argv) != 2:

        print("Syntax: %s filestem \n  %s will look for filestem.pcapng and filestem-keys" % (sys.argv[0],sys.argv[0]))

        quit()

pcapfile = sys.argv[1] + ".pcapng"

keyfile = sys.argv[1] + "-keys"

if os.path.isfile(pcapfile) and os.access(pcapfile,os.R_OK):

        print("Processing %s" % (pcapfile))

else:

        print("ERROR: Capture file %s missing or unreadable" % (pcapfile))

        quit()

if os.path.isfile(keyfile) and os.access(keyfile,os.R_OK):

        print("Using keyfile %s" % (keyfile))

else:

        print("ERROR: Key file %s missing or unreadable" % (keyfile))

        quit()

output_csv = sys.argv[1] + ".csv"
print("CSV file %s will be overwritten if it exists..." % (output_csv))

If we get this far, both the capture file and its accompanying TLS keyfile are present. We're ready to set a few variables, invoke tshark and start parsing its output:

stats_list = list()

tshark_cmd = '/usr/local/bin/tshark -2 -o tcp.check_checksum:FALSE -o ssl.keylog_file:' + keyfile + ' -Y "http.time || http.request.full_uri" -T fields -e frame.number -e http.request.method -e http.request.full_uri -e http.response_in -e http.request_in -e http.time -e http.response.code -E separator=, -r ' + pcapfile

p = subprocess.Popen(tshark_cmd, shell=True, stdout=subprocess.PIPE,universal_newlines=True)

Now to parse tshark's output into Python lists:

for line in p.stdout:

        line = line.rstrip()    # get rid of trailing newlines

        line = line.split(",")

It was at this point that my first test runs blew up in my face.  The .split method simply says "make this line of data a Python list, with commas delimiting list elements"...but I had forgotten that URIs can contain commas.  As a result, what I thought would be a simple Python list of 7 elements (some empty) in every case turned into Python lists of up to 79 elements when .split encountered commas in the URI!  So, I had to catch those cases and use an on-the-fly .join method to undo what .split had done to the third field of the line AND put literal commas back into the URI data...while leaving the first two fields and last four fields intact.  Only then could I append the (corrected) list to my master list-of-lists:

        if(len(line) > 7):

           line[2:len(line)-4] = [','.join(line[2:len(line)-4])]

stats_list.append(line)

(Yeah, figuring THAT one out took a few minutes.  *laugh*)

We're ready to match up request and response data, then write our CSV data. Here's the data structure:

# stats_list[x][0] = Frame number

# stats_list[x][1] = HTTP request method (only present in requests)

# stats_list[x][2] = Full URI requested (only present in requests)

# stats_list[x][3] = Frame number containing response (only present in requests)

# stats_list[x][4] = Frame number containing request (only present in responses)

# stats_list[x][5] = HTTP response time (only present in responses)

# stats_list[x][6] = HTTP response code (only present in responses)

...so, I used nested loops to search out the request/response pairs, then did a single write that pulled elements from both entries and wrote a single CSV line.  Since each line written represents a single HTTP transaction, I also counted them and informed the user of the total number of transactions found:

outputlinecount = 0

with open(output_csv,'w+') as out_file:

outwriter = csv.writer(out_file,delimiter=',')

for packet in range(len(stats_list)):

for target in range(len(stats_list)):

if(stats_list[target][4] == stats_list[packet][0]):

outwriter.writerow([stats_list[packet][0],

           stats_list[target][0],stats_list[target][5],

           stats_list[target][6],stats_list[packet][1],

           stats_list[packet][2]])

outputlinecount += 1

print("%d HTTP transactions written to %s" % (outputlinecount,output_csv))

The end result was a CSV file with entries like this:

6308,6315,0.125648,200,POST,http://www.foobieblex.com/cgi-bin/snarf

That's the packet number of the request, packet number of the response, elapsed time, HTTP return code, the HTTP method used (GET, POST, OPTIONS, etc.), and the URI requested.  (Remember those URIs with commas?  The .csvwriter method automatically quotes any fields containing commas, so that didn't bite me a second time.) This CSV file can be imported directly into any tool that accepts CSV data.  It isn't perfect - for instance, it doesn't (yet) catch HTTP requests that never completed - but it took less than an hour to write/test, and it's sufficient to the task at hand.  Here's a sample run against a 15MB packet capture containing roughly 25,000 packets:

Let me know what you think - or any Python tips/tricks to improve things - in the comments!

VIDEO: Where in the World Are Your Users? Geolocation with Wireshark

You've probably seen websites that greeted you with something like "Oh, you're in New York City? Here's our local store" or asked to "know your location".  If you've ever wondered how they do that, the answer is IP geolocation.  It's an interesting technique...and you can apply it to your own network capture data in Wireshark!

It's a neat trick; I've known mobile service providers who used it to create a dynamic map of locations they were "currently serving", and I've worked with data center operators who used it to create a dynamic heatmap of transaction loads from different parts of the world.  The best part is that - at least for simple, introductory purposes - you can start working with it for free!

In this video, I'll demonstrate how to enable IP geolocation in Wireshark, export the data in CSV format, and upload it to a mapping provider.  Basically, we'll go from a packet capture to a worldwide contact map in about 12 minutes.

As always - if you enjoy the video, please consider giving it a YouTube like and/or comment!

VIDEO: Decrypting End-User SSL/TLS Browser Sessions with Wireshark

Given that just about everyone is using HTTPS these days (and well they should!), troubleshooting web applications can be a major pain when it comes to network-layer analysis.  Fiddler is a solid tool, but its man-in-the-middle approach to capturing HTTPS sessions doesn't work in many secure environments, thanks to certificate issues.  What if you could just grab the end user's browser sessions and decrypt those?  Well, you can!

In this video, I'll demonstrate how to collect TLS session keys from Firefox/Chrome, import them into Wireshark, and work with the decrypted data.

If you enjoy the video, please consider giving it a like and/or a favorable comment...

Yeah, I'm Making Videos Now...

I finally took the plunge into creating videos.  Basically, I'll be creating short (less than 15 minutes) tutorials on various techniques I've developed in working with data networking and troubleshooting.

The videos are posted on IBM Collaboration Services' YouTube channel, ICSSupportVideos.

For those interested in such things, I'm using an off-the-shelf Logictech C270 webcam in a completely FOSS (free and open source software) environment:

• Platform: Ubuntu Linux 16.04.4
• Recording: SimpleScreenRecorder 0.3.9
• Network Analysis: Wireshark 2.6.1
• Still Graphics: OpenOffice 4.1.1
• Video Editing: OpenShot 2.4.1

Production values are minimal - I'm just learning this stuff - so be kind in your comments where aesthetics are concerned.  **chuckle**

If there's a particular topic in the networking world which interests you, let me know!

Synergy, Screen Savers and Security

My employer requires me to use a password-locked screen saver on my Windows 7 laptop, so I was surprised and confused last week when I unlocked my system to see a 'security violation' notification; my confusion became acute when I check my Windows configuration and found that I had configured the screensaver properly.

After a few minutes of experimentation, I discovered that Synergy was to blame. (For those unfamiliar with Synergy, it's a software-based KVM (non-video) system to share keyboard/mouse/clipboard across multiple platforms. While it was originally open-source, it has become a paid-download product - one of the VERY few such packages I use.)  As it happens, Synergy's default behavior is to synchronize screen savers across all participating systems; since my Synergy server (running Linux) did NOT have a screensaver enabled, the Synergy client on my Windows 7 system silently disabled its screensaver as soon as it made the initial Synergy connection.

Well, now that I know this, how do I fix it?  Well, there's some tweaking to be done on both sides.

On the Synergy server, I added the following section to /etc/synergy.conf:

section: options
    screenSaverSync = false
end

This disabled the synchronization of screensavers across my Synergy systems; however, I immediately found that I couldn't send a control-alt-delete signal to Windows 7 via Synergy.  Well, it turns out that, by default, Windows 7 doesn't allow the Secure Attention Sequence (SAS - the fancy name for control-alt-delete) to be generated by anything other than the system keyboard...but there's a fix for that. Head over to the Start menu and run gpedit.msc (the Local Group Policy editor). Navigate through Computer Configuration,  Administrative Templates, Windows Components, and Windows Logon Options. At this point, double click on "Disable or enable software Secure Attention Sequence" and configure the policy like this:

Once this policy was enabled and applied, everything worked as expected; my Windows screensaver kicked in when it should AND was password-locked, and I could send Control-Alt-Delete via Synergy with Control-Alt-Pause/Break. A quick security scan confirmed that my employer's requirements were met, and all is now well.

I'm not running Windows 8 or Windows 10 in my environment; if you ran into similar problems with Synergy, let us know in the comments.

Android-x86 and VirtualBox - A Potent Combination

For a "network guy", mobile devices can be really frustrating for one simple reason - unless you jailbreak the device, it can be rather difficult (if not impossible) to dive under the hood and get an idea of how the devices behave at the network layer.  Unless you just happen to have a spare device or two laying around and are willing to jailbreak them, you might be wondering if there's any way to observe network behavior in a fairly straightforward fashion. When it comes to Android, however, there IS a solution.

Now, it's certainly true that you can install the Android SDK and use its included emulator to run various versions of the OS, but that's a LOT of overhead; I don't need to dive THAT deeply into Android internals, and--to be honest--the performance of the emulator isn't all that great. I recently discovered the Android-x86 project, which has been going strong since 2009 to bring Android to the x86 platform. I installed Android-x86 on an old netbook and started playing with it, and then I realized...why not run it in a VM?

Enter Oracle VirtualBox.

This free virtualization package is available for Windows, OS X, Linux and Solaris; I'm currently using it on my Windows 7 laptop and several of my Ubuntu Linux machines, so I tossed an Android-x86 ISO into a new VM and went to work. Ten minutes later, I had this:

Android-x86 5.1RC1 running under VirtualBox on Windows 7

Now, it isn't perfect; since my laptop isn't a touchscreen, I can't work with gestures or multitouch, and (obviously) telephony functions aren't available. However, one can certainly exercise basic functions of just about any Android app within Android-x86...and, thanks to VirtualBox's extensive network support, it's a trivial matter to capture the network traffic of your Android VM with Wireshark. In no time at all, I was profiling the network usage/performance of various Android apps.

If you need to work with several versions of Android, Android-x86 can help you there as well; you can download ISOs of Lollipop, KitKat, Jellybean and Ice Cream Sandwich and install them to their own VMs. You can also share/copy VirtualBox VMs across multiple platforms (for example, I moved an Android-x86 VM from one of my Linux systems to my Windows 7 system with no problems). While I haven't done it myself, I'm told that some enterprise admins have registered their Android-x86 VM with their mobile device management (MDM) products of choice for use in testing/prototyping...

(NOTE: You can install Android-x86 to a bootable USB stick, if you so desire; here are the details.)

(NOTE #2: If you have a touchscreen laptop sitting around, give it a shot! Here's a video of Android-x86 4.4.2 (KitKat) running on a Lenovo Y50.)

So, whether you're testing, developing, or just want to play around with Android without buying a device or jailbreaking your personal stuff, take a look at Android-x86 and VirtualBox; they make a good pair.

New Linux installs - What Are YOUR "Must Have" Apps?

I think that every Linux user has their own list of "favorite" apps which, for whatever reason, aren't included in the default distribution. Some of our choices may be driven by work responsibilities, while others make the list for usability...and it seems that most of us have at least one or two "just messing around" applications as well.

While I'm an 'occasional user' of several Linux distributions (Red Hat, SUSE, Linux Mint, and Fedora), I'm currently running Ubuntu 14.04 and 14.10; I make no guarantee that the applications I list are available for every distribution, or that the release offered in your distribution is the most recent; I've provided links to the home pages of the various projects, in case you want to run the latest-and-greatest stuff. Having said that, and in no particular order, here are a few of the apps I automatically install on any new Linux box:

Shutter: As a software support engineer and networking geek, I use screenshots on a near-daily basis - lots and lots of screenshots, particularly in chat sessions with my colleagues. Shutter is a comprehensive screenshot tool which includes the ability to save images in all major formats, export to sites like Imgur and Dropbox, annotate/edit images, and more.

Wireshark: The opening screen of Wireshark reads, "The World's Most Popular Network Protocol Analyzer"...and, well, they aren't talking trash. If you're doing anything interesting at the network layer, you NEED Wireshark. It not only does network captures, but also reads/writes the file formats of every major network analysis device out there. (If you're running Ubuntu 12.04, 14.04, 15.04 or 15.10, there's an official wireshark-dev PPA you can use to install the latest build (Wireshark 2.0-stable) instead of building from source code!)

Docky: OK, I'll admit it - I don't like the Unity Launcher provided by Ubuntu. Docky is a nice, clean application dock/launcher that includes a selection of useful docklets/helpers; it's almost trivial to customize Docky to your taste.

GIMP: (GNU Image Manipulation Program) This is my "just messing around" app; I like playing around with images, even though I'm not very good at it (just yet). GIMP has tons of features, including many into which I have not yet delved, but it certainly does everything I need to do as far as image maniuplation is concerned.

BOINC: I very much like the idea of digital philanthropy; if my box has idle time, why not donate it to a good cause? Well, the Berkeley Open Initiative for Network Computing (BOINC) client is a good way to go; there are dozens, if not hundreds, of scientific research projects to which you can donate your system's idle time via the BOINC client. My personal preference is World Community Grid, which has my systems currently working on genome sequencing and attacking the Ebola virus. Since I keep several of my systems running 24x7, there's lots of idle time while I'm sleeping - 'nuff said. (Personal request - if you decide to participate in WCG, use this link to register...and that widget on the right sidebar of this blog will pick up a "recruitment" badge. **grin**)

That isn't my complete list of "must have" apps, but it's a good start - feel free to add your favorites in the comments!

Wireshark, GeoIP and Checking Up on Mobile/Home Carriers

As enterprises move an ever-growing list of services into the mobile space, it becomes essential to understand the limitations of the mobile network infrastructure.  No longer can we perform true end-to-end capture or analysis of network data; what was the "last mile" is now an indeterminate path through any number of relatively impenetrable mobile networks.  In this respect, troubleshooting issues involving mobile devices can be quite the challenge. At the same time, we're dealing with an increasing number of telecommuters, those "work from home" people who are at the mercy of their ISP.  What, then, is the enterprise network analyst to do?

The answer (or, at least, a good start toward an answer) lies in geolocation - the association of IP address spaces with their geographic and/or corporate assignments. Geolocation can be been integrated with DNS (or, at least, BIND implementations of DNS), the Apache web server, and any number of other applications, including (as of version 1.1.2) our favorite network tool - Wireshark. The marraige of Wireshark's analysis and GeoIP's provider identification produces some powerful analysis capabilities.

You can download free GeoLite versions of current GeoIP databases from MaxMind.  MaxMind provides free GeoLite databases for IPv4 and IPv6 city, country and autonomous system numbers (ASNs); you'll want to download the binary versions, not the CSV editions.  The MaxMind databases are updated on a monthly basis; if you like the results of this exercise, you'll need to set up a process to handle monthly updates.

Now it's time to make Wireshark GeoIP-aware:

1) Once you've downloaded the GeoIP databases, unzip them to a permanent home. On my Linux systems, I created the /usr/local/geoip directory for this purpose; on Windows systems, I use a \geoip subdirectory under the Wireshark installation directory.  The databases can be (and should be) read-only; you won't be adding any data. Now, we're ready to pull them into Wireshark.

2) Open Edit->Preferences in Wireshark, select Name Resolution, and click the "Edit" button next to GeoIP database directories; click New in the resulting dialog and add the directory you created in step 1. Using my Linux example above, you should have something like this (click to enlarge):

3) Close Wireshark and reopen.  You're ready to go!

So, what exactly does this give you? Well, to start with, you'll find that Wireshark's Statistics->Endpoints includes sortable columns for City, Country and AS Number, like so (click to enlarge):

You'll also find GeoIP information in the Details pane of the packet view, under Internet Protocol:

Finally, you can now use GeoIP information in your Wireshark display filters. For instance, I'll take the ASN definition from the previous example (British Sky Broadcasting) and use it in a display filter to show me ALL traffic from that provider:

ip.geoip.src_asnum == "AS5607 British Sky Broadcasting Limited"

Other GeoIP display filters allow you to select/view traffic based on country (e.g. ip.geoip.country=="Egypt"), city (e.g. ip.geoip.src_city == "Birmingham, AL") or even longitude or latitude (e.g. ip.geoip.src_lat == 33.520699).

From here, you can isolate, analyze and/or export data for specific providers, whether they serve mobile or home users; you could even develop "country profiles" if you're serving an international clientele.  While the GeoIP data isn't perfect, it's more than adequate to help you create a profile of your mobile userbase.

Have fun!

Network Troubleshooting - Sometimes It's What You DON'T See...

I spend a healthy chunk of my typical work day analyzing network packet captures.  My primary tool is Wireshark, which humbly presents itself as "The World's Most Popular Network Protocol Analyzer."  (Seriously - if you aren't using Wireshark, go download it NOW.)  Protocol analyzers are great for identifying typical "red flags" in packet data, but they're all limited to what the raw data might indicate; customer network environments are so broad (and so varied) that the network engineer--especially one "on the outside looking in" with only a small data set--relies heavily on experience and intuition.

One recent case was presented as "many failed connections," and a 6-minute packet capture soon landed in my lap.  Now, every Wireshark user has their own approach; I usually take advantage of Wireshark's display filters to get a general "feel" for the incidence of Layer 3/4 problems. With a typical capture file, I'll start with tcp.analysis.flags,which simply tells Wireshark, "hey, show me what YOU think are TCP problems." Now, as I said, none of these tools are perfect, so take these results with a grain of salt; they're only as good as are the underlying data, and it's very easy to collect inaccurate or incomplete data. After taking a look at the results of this display filter, I noticed what seemed an high number of TCP retransmissions, so I decided to see exactly which packets were being retransmitted with a different display filter, tcp.analysis.retransmission, which will show me only those packets Wireshark believes to be TCP retransmissions. The resulting numbers were somewhat high, but I've seen worse. Now, the complaint was very specific that new connections were failing; no mention was made of existing connections being interrupted/terminated; so, I went to Wireshark's Statistics->Conversations dialog and sorted on the "Packets" column to look for very short conversations and found HUNDREDS of conversations that only lasted for a few packets, like these:

Well, now, wait just a minute - the TCP handshake requires 3 packets (SYN, SYN/ACK, ACK) to establish a conversation, and I'm seeing hundreds of conversations that are only exchanging 3 to 6 packets. After checking a few suspect conversations, I found a pattern, namely this:

So, the remote endpoint starts a conversation with a SYN packet and the local endpoint responds immediately, but we see the remote endpoint retransmitting its SYN packet within 10ms. The local endpoint retransmits its SYN/ACK, but neither the original nor the retransmitted SYN/ACK seem to reach the remote endpoint, and the conversation attempt is ultimately terminated with a TCP reset (RST) packet. Back I go to Wireshark's display, this time to ask about a very specific type of TCP retransmission:

tcp.analysis.retransmission && tcp.flags.syn==1 && !tcp.flags.ack==1

With this display filter, I'm asking Wireshark to show me all retransmitted SYN packets; the "!tcp.flags.ack==1" eliminates SYN/ACK packets from the display. The results were startling; within a 6-minute period, more than 110 endpoints had retransmitted more than 170 SYN packets...and all of them had failed to complete the TCP handshake.

Well, if conditions are this bad to START conversations, then there must be thousands of cases in which existing connections die before completing successfully, right?  Let's go back to Wireshark's Statistics->Conversations dialog and sort on Duration to look at long-lived conversations:

Hmm...I have hundreds of conversations that last longer than 2 minutes...but I can't find one that suffers from retransmissions sufficient to terminate the conversation.

If I were looking at a general network congestion issue on the local network, I'd expect conversations to suffer equally--packets are packets, right?--but this is something different. That seeming conflict in the data prompted what proved to be the key question:

If I'm seeing HUNDREDS of new conversations fail the TCP handshake due to excessive retransmissions, why DON'T I see established conversations suffering excessive retransmissions as well?

Well, after few moments' thought, it occurred to me that the only network devices that usually make specific distinctions between new and existing connections are those involved in network security. A brief conversation with the customer revealed that an intrusion protection system (IPS) was in place and "inspecting" conversations. When we conducted a test that bypassed the IPS, the incidence of failed TCP handshakes decreased by roughly 98%; our troubleshooting attention is now properly directed.

So, the moral of this story: Pay attention to the data, but pay equal attention to what isn't there.

SxSW = Tons of Free Music!

Austin, Texas is cranking up South by SouthWest 2014, and that means music - LOTS of music. SxSW gets most of its mainstream press for its technical and creative content these days, but the festival's roots are musical; hundreds of bands will be playing in dozens of venues. (Can you tell that I REALLY wish I were attending?) As you might expect from a festival that features so much indie music, there are more free samplers and promotional releases than anyone could possibly want to download...unless, of course, they're a total music geek. So, since I'm already downloading these (grin), I thought I'd share the links with you. In the interest of brevity, I won't go into lengthy descriptions of each collection; the only thing I'll say is that I inevitably find REALLY good music, across a range of genres, in every year's samplers. Having said that, let's get to the list: 

NPR's Austin 100 - A massive (839 MB) collection of NPR's picks. 
Arts & Crafts Records - This Canadian label will have a healthy presence in Austin. 
Bar/None Records - Always an interesting selection.  
SxSWBaby! - Another large (80 tracks, 314MB) collection from the thoroughly UNofficial SxSW blog. 
Rubberneck/Burger City - Truth be told, this one is brand-new to me; I'm downloading the tunes as I write this...  
Dine Alone Records 
OK! Records 

Some of these are straight downloads, while others require your email address. Enjoy!

#SpeakerU - Speaker Coaching: It's Worth The Time

OK, so, you've been selected to speak at a conference.  You've spent hours building your slides, you've gone through your outline a dozen times, and you've even jotted down particular words (or turns of phrase) you want to use.  There's only one problem - you've never really given this presentation to an audience.

Enter the speaker coach. 

Now, everyone seems to have their own idea of what a speaker coach should be; as far as I'm concerned, a good technical speaker coach should be able to do two things:

• Critique the mechanics of your presentation style, and
• Evaluate the technical content of the presentation itself.

The former category can be tricky for technical folks.  It seems that we're used to people speaking fast, using run-on sentences, and throwing jargon about with reckless abandon.  Most of us have to take a breath and slow things down a bit when speaking to an audience that isn't as familiar with the subject matter as we might be.

The latter focus--the technical content--should be evaluated in the context of the intended audience.  If you're targeting your presentation to beginners, the question is, "Did they get from [nothing] to [basic understanding]?"; presentations to folks with intermediate skills should result in the listener "getting to the next level," and speaking to advanced audiences...well, at that point it's usually about bringing something new to the table, since they (by definition) already know the beginning and intermediate stuff, right? So, you want to find someone who fits the "typical attendee" model, NOT someone who already knows the material.

Long story short, folks - find a person (or persons) who can do these two things for you.  Take them through your presentation.  Don't stop, don't pay any attention to the fact that there are only one or two people in the room, and don't hold anything back: give the presentation as you expect to deliver it on stage.  If you can get a projector, do so; make this as realistic a "dry run" as possible.  After you're done, sit down with them and ask two questions:

1. "Could you hear and understand my words and how I spoke?"
2. "What can you tell me about [subject of presentation]?"

Their answers will tell you what (if anything) you need to change.

I can't say this strongly enough: DO THIS.  DO THIS SEVERAL TIMES.  Every runthrough you can make before hitting the stage will improve your performance.

Now, go practice!

#SpeakerU - Building Your Deck, or "You Want Slides With That?"

Previously, on #SpeakerU...

• Time to Get You on Stage 
• First Step: Picking A Topic
• Writing an Abstract, or "PLEASE PICK ME!"

So, you sucked up your courage, submitted a proposal to a meeting or conference, and then--O happy day!--you received that wonderful "your abstract has been accepted" message.

Now you must...MAKE SLIDES.  You may need to make MANY SLIDES.

Here's where you really have to start thinking about the mechanics of your presentation: how much time you'll have, what points you MUST cover, your own comfort level with the information, how much time to spend on each piece of information...all of this goes into the slides you're about to create.

At this stage of the game, one's slides often take on the 'personality' of the speaker.  This is, I think, as it should be.  The goal of #SpeakerU is not that everyone who reads these articles churns out presentations of identical style--that would result in really boring conferences, if you ask me--but rather that your slides balance your speaking style with the material you need to cover.  Remember, those folks are coming to hear YOU; if they only wanted the raw information, they could have stayed at home and read a book.  Find that comfort level that lets you feature the content.

I'll also recommend a visit to Slideshare, where folks from around the world have posted presentations on a wide variety of topics.  There's nothing wrong with looking at the different formats and styles on display to find your way, but resist the temptation to copy slides wholesale.  (You WILL be caught eventually; remember that the attendees can check out Slideshare, too...)

Instead of going through the mechanics of building slides or talking about point sizes and the like, I'll leave you with some general guidelines I use when laying out the slides for a presentation.  Feel free to use or adapt them as you see fit; your deck is about your style and your information, not mine.

So, then, in no particular order:

• Assume that your audience can read.  Reading your slides is THE fastest way to lose your audience.
• Make a noticeable transition, either vocally or by changing slides, every 90 seconds or so - or you'll start losing people.
• Resist the urge to use copyrighted material of any sort.  Imitiation is the sincerest form of flattery, but it's also the fastest path to a conversation with lawyers.
• If you can't cover a one-topic slide in less than 2 minutes' time, split the information across multiple slides.
• Only use effects/animations when they deliver a necessary accent or transition to the content - in other words, forget the eye candy.
• Don't be afraid to use a (well-designed) graphic instead of words, but remember to give enough written information to spark the listener's memory when they read your slides a month or two later.
• Don't "get fancy" with language. You'll be seen as a show-off, especially where words/phrases from other languages are concerned.
• Remember that you will be "talking AROUND your slides," adding perspective/depth to what your audience is reading.
• Don't let your slides "skip around" - don't take your audience to Point C if you need to talk them through Points A and B first.
• If you use graphics, remember that they need to be clear to the folks in the last row - avoid clutter.
• Be VERY careful with humor, especially when delivering to an international audience.
• Don't create eye charts; use a reasonable font/size for the projection system that will be in use, and stick with it.

Once I've made a "rough draft," I usually put it to two tests; I hand it to someone who knows the topic well ("Did I miss anything?"), and then hand it to someone who doesn't know the material at all and ask, "Can you follow the general flow?"  Remember that your audience will probably cover the spectrum from 'newbie' to 'expert;' you want to inform the former without losing the latter.

Next up - putting it all together...

As always, feel free to add your own experiences, tips, or pasta recipes in the comments...

#SpeakerU - Writing an Abstract, or "PLEASE PICK ME!"

It's time to get back to #SpeakerU, our ongoing discussion of how you can "get up on stage" and become a technical presenter/speaker at conferences like IamLUG (now ICONUS), ICONUK, and IBM Connect.  (If you want to play catchup, my previous #SpeakerU articles are here and here. Several other folks have written articles, as well; a quick search for "#SpeakerU" should bring their work to your attention.)

Most conferences do NOT expect you to have a full deck of slides (or a complete paper, or a finished poster) prepared in advance; instead, they'll ask you to submit an abstract - a brief explanation of your topic and material.  Now, whether you realize it or not, many conferences make the abstracts of accepted sessions public well before the conference meets, so that prospective attendees can begin planning their schedules.  Thus, your abstract has to do two things:

• Convince the folks hosting the conference that you know your stuff, AND
• Convince attendees to attend your session.

Your task is complicated by the fact that abstracts are usually limited in size; you might have as few as 500 characters with which to "make your pitch."  (Remember in the last article, when I said that a big part of picking a topic was fitting it to the time available?  Well, here's another reason to choose wisely; you have to be able to explain it succinctly!)  So, how are we going to accomplish so much in so few words?

I tend to use abstracts to do one thing - ask/answer attendees' questions.  If you've identified where folks are having problems AND can deliver solutions, that's going to be a big win with the conference organizers; if you hit upon a common problem (or a Frequently Asked Question), you may hear a buzz about your presentation well before the conference and (eventually) face a packed room.

Instead of creating spurious examples at this point, I'm going to point you to Russell Maher's recent article on the subject.  He gives the best possible example of the abstract process - his own abstracts. (As you'll see, I collaborated with him.)  The key point is take your topic and put it in terms of the attendees' environments, their questions and your answers; remember that they may not even realize that they need to visit your session until they read your abstract.   It isn't what you can teach - it's what they can learn if they attend.  Russell provides a before-and-after view of two session abstracts, which I think illustrates the process pretty well.

For all this talk about 'selling' and 'pitching' your session/presentation, it's important NOT to oversell.  Remember, you can't claim to make attendees experts in a one-hour session, nor can you ever answer "all the questions."  Be reasonable and straightforward about what you can deliver; as I've often said on stage, sometimes the best outcome of a presentation is NOT that everyone finds their questions answered, but rather that they learn the right questions to ask!

A good abstract requires effort; it isn't something you dash off in 90 seconds.  If you can find old conference programs, take a look at successful abstracts from past events.  Ask your friends to review your first effort, ask others to rewrite them, and don't be afraid to take it through several iterations.  The important thing is NOT to give up - keep trying!

Next up: You've Been Chosen - And Must Now Create Slides...

Puzzling Through Music? Here's My Chordspeller...

If you've been paying attention to this blog (or if you've ever met me), you know that I'm a complete music nerd.  Well, that extends to music theory; I'm one of those folks who fakes playing piano by cheating from the guitar notation, and I'm one of those singers who wants to know whether I'm singing the root, third, fifth or seventh of that big chord at the key change.  Well, to do that stuff one needs a fairly comprehensive chord library, whether it be a mental storehouse or a cheat sheet.  I'm not that good - I need a cheat sheet.

Long ago, when I first started dissecting barbershop music (and, yes, baritones DO have the strangest part!), I had a great 'chord spelling' chart; unfortunately, it was lost some years ago and I've never been able to find a good replacement.  My daughter recently ran across some 'fake sheets' for current pop songs, which have naught but lyrics and guitar chords; she started asking me "What's a Cdim7?  How do you play an Fm9b5?"  I thought of my old chord speller, and decided to create my own.

Simply put, it's a one-pager that includes:

• Nomenclature (e.g. dominant 7th, minor 6th, etc.),
• Symbol notation (e.g. Cm7, C+, etc.), 
• Scale note formula (e.g. C minor triad = "1 b3 5"), and
• Chord listings (e.g. "Gbm9 = Gb A Db E Ab")

for 204 chords, plus extra notation & scale note formulae for another 15 chord families, for a total library of 429 chords - on a single 8.5"x11" page.

You can grab a PDF copy of Chordspeller from my Slideshare library.