Realized my last post was almost two months ago! It’s been a bit of a busy period.

Clementine shared balloons and fun with kids at a community festival, marched in a parade, and served as a entertainment on the midway. Oh, and got herself a new pair of polka dotted oversize Converse sneakers!

I bought, built, and directed the decorations for vacation bible school, led games for vacation bible school, and went with a group of youth (including both of my daughters) on a week long mission trip. I even got to use one of my clown magic tricks for the VBS kids… that was my excuse to buy the requisite parts, so good to use it at least once or twice.

In between, I’ve also been working my way through a Coursera multi-class specialization on strategic leadership, and gotten to help bring a few candidates into our company. A bit more learning on Kubernetes, Gitlab, and ELK as part of the day job, too.

Oh, and we got a kitten! Strategy suggests I should keep up with my daily allergy pills. Miso (kitten’s name) is cute, but he makes ‘me so’ sneezy

I need to be at work earlier this week than normal. We’re giving training on Docker and Kubernetes, and the first class session starts at 8. The classroom needs to be open earlier than that, and I want to be in and settled before the first student arrives.

That’s meant that I’ve set up a whole system of wakeup alarms, between my phone and my alarm clock.   The phone has multiple staggered times, and the clock has another one.  That’s to handle the challenge of sleeping through or turning off an alarm and remaining in bed.  But that means that once I’m up and moving, there can be multiple alarms that still go off. Did I mention that my husband _doesn’t_ need to get up earlier this week?

I’m now thinking through my optimum user interface to let me say, I’m up, keep the rest of the alarms from going off.  Is it: no more alarms at all today?  Is it: no more alarms within the next two hours?  Is it: show me a table of the alarms yet to go off this morning and let me turn them all off in one quick move?  Is it: here are my wakeup alarms – turn them all off, yet leave reminder alarms for later in the day?

Oh, and I’m hunting for the source code for the existing Alarm clock app and widget.  The app has an option to let me see its open-source licenses (Apache, MIT).  That bodes good things for getting to try something out..  I’ve only done minimal Android development before, just enough to make a UI to control a Furby.  I like the idea of solving a personal concern and getting to play in geekery.  Now we’ll see if liking the idea actually translates into hours spent sometime.

Some stuff for work is running on a Kubernetes 1.3 cluster.  It nicely restarted when something went wrong in its environment.  That’s good.  Our pods were logging – that’s good.  But when the pod restarted, its logs were lost: that’s bad.  No idea why the pod restarted to help determine if it’s something we should either prevent or add a health check for.  So, now I’m evaluating logging solutions appropriate for our Kubernetes environment.  And because I’m a geek, I’m blogging about it here in my pjs on my day off: you know you’re in the right field when you’re enjoying researching something for it on your day off.

Scope of problem: not trying to log all of the kubernetes events themselves: that’s the job of the infrastructure.  Which also means then I’m not worried about filtering out messages I shouldn’t see from pods that aren’t my own.  I want to collect log messages in my namespace, have them tagged (at least) by which application source they came from, and I want to have them be archived _off_ of the cluster.  The guys running the infrastructure are great, but if they reconfigure the cluster (as happens occasionally) or have a hiccup with their persistent volumes, still want my logs.

A bit of survey suggests there are two primary solutions out there: Fluentd or Logstash, one of the components of the ELK stack.  Both are open-source.  I’ve long heard of ELK, though not yet myself futzed with it in a system.  I’ve seen many stickers for Fluentd in open-source conferences…  :-)  I’ve also noticed Fluentd as part of the Cloud Native Computing Foundation‘s list of supported projects.  Hmm..  that last bit gives me a bit of bias towards Fluentd, sight unseen.

So now to look for other folks comparisons of the two to pick the first one to try out..  My strategy is to start with one and see if it’s ‘good enough’ to meet our needs.  But I’ll often rely on what the ecosystem has already discovered to make my choice between.

Fluentd vs. Logstash: A Comparison of Log Collectors: logz.io uses both.  My takeaway – leans towards Fluentd in its recommendation, even though it’s built on ELK

Fluentd vs. Logstash: Panda Strike’s tried both..  They’ve ended up with EFK (ElasticSearch, Fluentd, Kibana) as their platform for reasons of stability and performance.  Side note: describes logs forwarders vs. log shippers.  Found another post on their site that describes EFK in action for JMeter load testing result visualization.    Another side note: found a good bit of useful content in their blog overall..  adding ’em to the rotation of stuff to keep an eye on…

Fluentd vs Logstash: Jason Wilder walks through both.  Older post (Nov 2013) but assuming all is still relevant, good technical info.

Outcome: looks like I’ll be starting with Fluentd.

 

 

 

 

I got a cold-call from a recruiter today, seeing if I’d be interested in a Technical Manager job.  That’s right in my bailiwick: I tend to be more technical than most technical managers, but she had me pegged in the right general zone of job, anyway.  What surprised me was her saying she had a resume of mine from 2006.  Wow.

Luckily for me, 2006 rang a bell.  That’s the year my son was born, so that’s the year I traded a commute around the Beltway to a job I loved for a new series of adventures with a company closer to home.

In 2006..  I was building an e-commerce system using Struts.  I _think_ we were debating a rewrite in Struts 2.  Somewhere in that time frame, I became a certified Scrum practitioner.  (I’ve since let the certified part of that lapse, though I’d still consider myself a Scrum practitioner.  Just one who doesn’t pay certification fees to prove her worth.)  We’d do production deployments once a week, beginning at 11pm and working through the night until we succeeded or rolled back.

Since then, I’ve done all sorts of things…  OSGi in its relatively early days.  Grails web applications that I helped take GOSS (government open-source) and then fully open-sourced.  D3 visualizations for SBIRs.  Actually, D3 visualizations for two different SBIRs.  Many hours realizing I needed to better grok D3.  Lots of business development.  Winning multiple proposals and then staffing teams to do the technical delivery.  Where those teams typically included me, hands-on coding.  Ramping up on AngularJS and Node to lead a “thin-client” team.  That’s as opposed to the program’s thick-client team building on top of Java Swing – thankfully, although I managed the overall UI team, didn’t have to wade too far into the Swing code.  Docker containers containing stub implementations of enterprise system interfaces we wanted vendors to be able to execute against.  Now Kubernetes: figuring out how to deliver and sustain systems in a cloud.  I’ve delivered two OSCON briefings, one using an Arduino and C and one using Go and a Raspberry Pi.

It’s been a fun 10 years.  If she calls again, though, she’d better go find my updated resume.  For the record, LinkedIn’s got a pretty good idea of what I’ve been up to…

A few months ago, I joined a software team that’s making use of Kubernetes as a platform to provide elastically scalable software services.  It’s Docker + more magic.

One of the tenets espoused by some folks building container deployments is that a container should only have the software on it required to run a single process / microservice.  The particular software system I’m deploying is a private Docker registry, but to deliver the service, there are currently 4 Docker containers in play.  (I actually think that to solve a problem I’m having, I’ll need to add two more.)

One of the containers wasn’t behaving nicely for me today.  The container provides a web user interface and needs to communicate with a database provided by another container to store and retrieve its data.  But the logs were saying that the user interface container couldn’t log into the database.

The first two options that came to mind:

– the hostname wasn’t accessible to the ui container

– the port wasn’t open on the database container

Turns out, it was neither of those: it was a badly shared database password.  But in trying to prove or disprove those options, I ran into a challenging problem.  See, the UI container _only_ had the binary for the UI process on it.  Which meant: it didn’t have ssh, telnet, curl, wget, netstat, or any of the various other options I came up with to verify network connectivity between the containers.  How to get my answer?  Google + Stack Exchange to the rescue!

In recent versions of bash (which thankfully the golang image did provide), bash will let you access TCP and UDP ports directly, ala

cat < /dev/tcp/databasehost/3306

Running the above from the ui container got me a garbled prompt received from the database container, which proved that the hostname resolved AND the port was open, all in one command.

Kudos to Peter Mortenson and his answer on superuser.com  for helping me crack the bug.  Tuesday’s bug (courtesy of the Monday holiday) will be to see if Beego.Run() doesn’t crash if I manage to map in the app.conf file not otherwise being provided through my Kubernetes replication controller descriptor.  Not good to have applications just stop without an error message!!

 

 

We’re hiring at my company, Hayden Software, and I’ve been asked to help interview for my contract, so I’ve had a few different opportunities to ask candidates how they track down programming problem solutions.  The usual answer?  Either Google or Stack Overflow.  Both are reasonable answers, though are a bit broad brush in terms of their coverage.  If I find something mentioned in documentation, what I’d _really_ like is to see a working example of it.  Particularly since most programmers seem to write awful documentation.

I’ve taken, of late, to trying to use GitHub as my compendium of source code examples.  I’ll assume that the folks who cared about this post already know what GitHub is…  if you’re hear to read a post about clowning, you’ll have to wait for the next one…  GitHub announced in 2013 that it had reached over 10 million code repositories.  It’s not quite yet the infinite monkey theorem, but there’s definitely enough source code there to make me think that any library or framework feature that I’m interested in using already has a bit of sample code for it in someone else’s repository.

But how to find the examples I’m seeking?

Recently I was seeking examples of fabric8’s docker-maven-plugin in use.  It’s one of a couple of plugins that are supposed to help with managing Docker builds and Docker containers.  I don’t personally find Docker build files particularly challenging, but perhaps the managing the containers part would somehow be made easier with the plugin.  (TL;DR: finally didn’t find it useful, at least for our uses).  But in my hunt, I wanted to find examples of folks using this thing…

That plugin would be used in a pom file.  So, I want to find pom files that make use of the plugin.  In the case I originally went looking for, I wanted to use something called inline plugin configuration.  Basically, if I was going to use a Maven file to do the docker build, I wanted to keep all of the build information in the single pom file.  The docs described it as possible, but it just wasn’t coming together for me.  So, the query against Github:

(inline assembly docker-maven-plugin fabric8 dockerfile)  filename:pom.xml fabric8

Look across all repositories for a file named pom.xml which had certain keywords in it, and which were part of a repository or owned by a user which included the word fabric8. Found a few examples (and, and), but nothing which blew me away on the usefulness of that plugin and approach.

What about when I wanted to see examples of a particular routing package used in Go code?

language:go beego router

Wow, that gave me back 7K+ results.  Maybe narrow that down a bit?  I’d like to narrow that down by cross-checking the stars on the owning repository: theoretically, something with more stars would have more useful examples.  Interestingly, I cannot seem to get that to work as I’d like…  Although I can hunt for repositories with the phrase beego in their name or referenced in their description, I cannot get it to further check files.  The query below ignores the ‘stars’ descriptor and gives the same 7K+ results.  (I say ignores because I pulled up one of the repositories and it showed zero stars.)

language:go stars:>10 beego router

So, GitHub filtering has its challenges and quirks.  That said, I can also filter on looking for code within a particular repository or by  particular user, both of which help in filtering out the wheat from the chaff.

I can’t learn everything in software on my own, but if I can learn from someone else’s example (good or bad, assuming I recognize the difference), I can get the thing done and build upon my knowledge very efficiently.  Would love to hear if other folks have code search tricks and techniques, on GitHub or otherwise.

 

Seen on Twitter this morning with guilty amusement:

 

Made me laugh as I thought of blocks put into my Docker files to let me go in and see what state things were in after I ran them.   Move 2 lines of code around from place to place and see what new info I glean…

#####DEBUG - exit here to check state
exit


Or console.log statements with odd text that should _never_ make it into checked-in code, but acts as a visual trace of the flow of code.

Or conditional asserts put into code to verify that a condition really does happen on the Nth +1 iteration of the loop.  (Handled with less danger of an inadvertent code check-in by setting these up as conditional breakpoints in a debugger..)

Or even comments added in to help us better understand what we think we’d like to keep versus toss in a burning hole.  (Source of comments: Apollo 11 Guidance Computer source code…  Way before my time, but demonstrating another ancient programmer spell…)

Got any new tricks to add to my bag?

Ran across a comic strip this morning described as “9 Non-Threatening Leadership Strategies For Women”. . Linked to the original source, but the reminder of it this evening came from Fast Company’s “How to be a Non-Threatening Woman”, which linked back to the Cooper Review’s original article…

In this fast-paced business world, female leaders need to make sure they’re not perceived as pushy, aggressive or competent. One way to do that is to alter your leadership style to account for the (sometimes) fragile male ego.

When I first skimmed it this morning, I thought the writer was trying to give good advice… “Trying” being the important word here. I grew frustrated as I read the article, until my caffeine-deprived brain got enough clues that this was a tongue-in-cheek approach and meant to highlight the pain-in-the-rear-edness behaviors some of us experience, and the stereotypical coping strategies / highly annoying behaviors some folks use to cope. (Note that I’m not focusing solely on women doing the coping – seen guys do some of these things as well. Just as highly annoying.)

I’m a woman in tech. Highly outnumbered and have been throughout my career. On my current project, in a room of 35, I’m one of 3 women. The 3rd woman joined one month ago, while the 2nd woman was still on maternity leave. So, for several months, just me in the estrogen column. Neither of the other two women are leaders of groups in the room. Leaving me the sole female leader in a room where the guys shoot rubber bands at each other as a form of male bonding, where another of the tech leads said his frat-hazing treatment of a more junior member of the team was his way of showing care, and where one team member gave another leader in the room a thong as a way of poking fun at his accent in pronouncing ‘thon’… I’ve been talked over, had things demeaningly explained to me, and had my points and ideas repeated by others (and thus finally heard)… I definitely lean to the ‘left’ on the pictures presented in the article, and refuse to shimmy to the right.

Next week, I leave that particular team. I didn’t think I was leaving because of the sort of behaviors demonstrated – there were other reasons to leave. But suddenly realizing how much I’m looking forward to gelling with a new team.

In something of a motivation slump at work. The end’s in sight, it’s just farther away than I’d like. So I sit here on a Wednesday morning weighing the many productive things I could be doing today, instead of heading to work.

    I could be…

  • Working on reviewing my Gobot presentation for Golang Maryland tomorrow evening
  • Moving further with my Apache Spark edX class. I did some of it this morning, and am just at the point where I can play with notebooks and actually futz with things…
  • Spending more time with ‘The Children’s Illustrated Guide to Kubernetes’, which I tripped across through a Twitter mention this morning. Kubernetes will turn out to be useful soon-ish, so trying to absorb info, concepts, and practical tutorials
  • Revising my clown makeup approach – doing a festival this weekend, and didn’t like how my last facepaint worked out
  • Practicing balloon animals for that same festival
  • Visiting a friend in a nursing home to try getting her wheelchair in my vehicle ahead of our Orioles game this weekend

List is long, and so rewarding. But if I let myself lapse on going into work today, I’ll want to do it again tomorrow.

If you give a software engineer a free day…

If you Give a Mouse a Cookie