Wednesday, August 15, 2018

Speed Scripting - Making an Animated GIF with ImageMagick

Those of you who follow me on Twitter probably know that I'm something of a geek-of-all-trades; I love reading and learning.  When my employer, IBM, entered the world of Open Badges for employee credentials, I jumped in with both feet.  We're encouraged to display our badges in email signatures, online, et cetera...but what is one to do when one has earned 50+ badges in a wide range of areas/disciplines?  Now, obviously I can't put that many badges in an email signature, and a static display can only do so the thought occurred to me, "I should be able to turn these into an animated image!"  Time for some shell scripting...

Let's consider the source material:

  • Our badging provider provides badges as 352x352 PNG images, so I don't have to worry about resizing individual images; I can treat them all the same way and resize the end product as needed.
  • I'm not worried about the PNG metadata in this particular case, so converting to another format isn't a big deal.
  • I could go with an animated PNG (APNG), but (a) not all browsers display APNGs properly, and (b) animating PNGs can result in images significantly larger than comparable animated GIFs

At this point, I'm ready to start playing with ImageMagick, a wonderfully complete (and open source!) image manipulation package.  ImageMagick handles 200+ image formats, and is available as both a command-line toolbox and libraries/interfaces for a wide range of programming languages.

The actual construction of the animated GIF with ImageMagick is straightforward - it's a one-liner:

$ convert *.png -frame 5 -dispose Background -set delay 150  allmybadges.gif

The options put a frame around each image, dispose of the previous image before displaying the next one, and set the delay (in hundredths of seconds) before adding them to the end product, allmybadges.gif.

Wait, though...oh, I don't like that.  Since many badges have similar names (e.g. task-level-1.png and task-level-2.png, ibm-iot-whatever1.png, ibm-iot-whatever2.png, etc.), my use of a wildcard puts them in alphabetical order...which is kind of boring when 2-3 consecutive badges share color schemes or other design elements.  So, what I really want to do is randomize the badges before animating them.  Well, bash provides $RANDOM as an environment variable, yielding pseudorandom integers between 0 and 32767; that should be sufficient for me to play with filenames.

I wound up with this script:

# badgeanim - create animated GIF of all PNG badges

cd $WORKDIR                          # move into working directory
rm $WORKDIR/*.png                    # blow away any old PNG files

for file in $BADGEDIR/*.png          # grab copies of current PNGs
cp $file ./$RANDOM`basename "$file"` # prepend $RANDOM to filename

convert *.png -frame 5 -dispose Background -set delay 150 allmybadges.gif                                         # ImageMagick builds the GIF

rm $WORKDIR/*.png                    # blow away leftover PNG files

The results can be seen at the top of this article and in the "My OpenBadges" widget on the right margin of the page.  (Clicking on the animated GIF to the right takes the viewer to a static display of all badges.)  I could resize the end product for use elsewhere, like so:

$ convert allmybadges.gif -resize 200x200 allmybadges-200x200.gif

Check out ImageMagick - it's definitely worth your time if you do anything at all with images.

Saturday, August 11, 2018

First Steps in node.js - Fortune Cookies!

I've been doing a lot of learning around cloud architectures, microservices, and the like, so I thought it was time to start learning JavaScript and node.js.  I started working with IBM's SDK for node.js, but soon learned that IBM has deprecated its proprietary SDK in favor of the community SDKs. I installed the latest/greatest community SDK (v10.8.0) on my Ubuntu Linux system.

Some months ago, I wrote a Twitter bot that delivers a random fortune-cookie quote/saying every 6 hours. (It's @CollectedQuotes, if you'd like to follow it.)  Well, what better example could I adapt as a simple service in node.js?  Here we go, short and sweet (click images to enlarge)...

Here's the source code:

Here's the (very simple) browser page it generates:
Here's the 404 response to any URL request other than the root:
Finally, here's the console log:

It might be clunky, but it works!  If you'd like to play with this, you can download the JavaScript source and the fortune-cookie file.

Wednesday, August 01, 2018

Optimizing Wireshark for HTTP Analysis/Troubleshooting

I spend quite a bit of time troubleshooting various web applications, so I've done a lot of work with Wireshark's HTTP display filters. The ones I use most frequently are:

  • http.response.code - In an HTTP response, the numeric response code (200, 404, 500, etc.)
  • http.content_length - In an HTTP response (and PUTs and POSTs), the total payload size
  • http.request.method - The type of request made by the client (e.g. GET, PUT, POST, CONNECT, etc.)
  • http.request_in - For an HTTP response, the packet # of the corresponding request
  • http.response_in - For an HTTP request, the packet # of the corresponding response
  • http.time - elapsed time between request and response

That last filter needs a bit of explanation, because it can be computed in two different ways.  If the TCP preference "Allow subdissector to reassemble TCP streams" is enabled, http.time reflects the time between the request and the last packet of the response (i.e. the end of any data returned); if that preference is disabled, http.time reflects the time between the request and the first packet of the response (i.e. the HTTP response code).  I almost always have that TCP preference enabled, because the client/browser usually can't do anything with the response until it receives all of it!

So, one could work through a Wireshark session, using display filters like http.response.code==404, http.content_length > 4096, or http.time > 2.0 to display various packets...but, to be honest, I'd rather not do that much typing.  So, I set out to optimize Wireshark's performance and tweak its display for HTTP analysis.

The most significant performance optimization one can implement in Wireshark is to disable analysis of irrelevant protocols.  By default, Wireshark tries to dissect every protocol it can identify...and there are hundreds of protocols in its dissection engine.  Disabling irrelevant protocols will greatly enhance overall performance.  Keep in mind, though, that one needs visibility at all layers of the network stack; for most work in my environment, I need Wireshark to dissect Ethernet, IPv4, TCP, UDP, ICMP, SSL/TLS, HTTP, and a handful of other protocols.  I disabled analysis of all other protocols.
Now, to the GUI tweaking.  First, I created filter buttons for general display filters, so that I can apply them with a single click:
  • http - apply http - display all packets identified as HTTP
  • 1xx - apply http.response.code < 200 - display responses with informational codes
  • 2xx - apply http.response.code > 199 && http.response.code < 300 - display responses with success codes
  • 3xx - apply http.response.code > 299 && http.response.code < 400 - display responses with redirection codes
  • 4xx - apply http.response.code > 399 && http.response.code < 500 - display responses with client error codes
  • 5xx  - apply http.response.code > 499 - display responses with server error codes
  • >2s - apply http.time > 2.0 - display all responses that required more than 2s to complete
  • GETs - apply http.request.method=="GET" - display all GET requests
  • POSTs - apply http.request.method=="POST" - display all POST requests
Then, I added columns for use in "eyeballing" HTTP traffic and doing quick sorting (you can sort on any column in Wireshark's display):
  • Stream - display, the connection identifier generated by Wireshark as the file is read
  • Req # - display http.request_number, the request's sequence number within its connection (useful for HTTP-pipelined connections)
  • HTTP Req - display http.request_in for HTTP responses
  • HTTP Res - display http.response_in for HTTP requests
  • HTTP Time - display http.time
  • HTTP RC - display http.response.code for all responses
  • Payload - display http.content_length for all requests/responses with data payloads

Here's a sample of the results (click to enlarge):

(You can see the "one click" filter buttons in the top right.)

From here, I can sort on any column (like, oh, HTTP Time?), match requests and responses easily, identify "red flags" at a glance (why on earth did it take 1.5s to pull down PNG files of only ~280 KB?!), and examine how "red flags" affected subsequent requests on the same connection (that 1.5s delay affected the browser's processing of the next GET as well, since they were on the same HTTP-pipelined stream), all from a single view...and Wireshark's processing time is greatly improved, to boot!

Now, this isn't perfect.  If any problem occurs in IP/TCP/SSL-TLS/HTTP reassembly (for instance, a missing or corrupted packet), you won't get full information on affected HTTP transactions...but it's easy to right-click on the "missing information" packet and use Wireshark's Follow TCP stream command to zero in on a transaction with missing information and determine what happened.

(If you'd like to try this profile, you can download it as a ZIP file.  You'll want to unzip it in your Wireshark profiles directory; it will create a profile directory named 'HTTP'.  Within Wireshark, you can switch profiles by clicking on "Profile: Default" in the bottom-right corner.  Remember, though, that any configuration changes you make are automatically saved to the current profile; be careful!)

So, that's the Wireshark environment I'm using for HTTP analysis.  Did I miss something you consider important?  Is there a change you would make in your environment?  Let me know in the comments...