Jan 10

objective measure


this ruler is deep and spiritual, that's why it was photographed in black and white

I’ve found myself in an odd situation as of late regarding my work, I am a programmer by trade and education but haven’t done any professional programming for 2-3 months now. It’s not that I’m being lazy and don’t want to program, I would jump at the chance to fire up Eclipse right now and start writing mountains of verbose Java code, it’s that I’m not allowed to program. “Not allowed?” I hear the shock and dismay in your voice, so let me explain.

My role on this project is to design middle-ware but because of the way the business is set up I’m not actually allowed to program, the programming is all outsourced. Instead of actual programming I get to engage in some weird form of meta-programming where I turn a program into long flowing sentences in a Word document that take about 3 times the effort to write than the program itself so that it can be shipped off and someone else can program it. It is an odd situation indeed, but I am no business man, this project is important, and so I shall continue in this weird meta-programming.

This is not my first dance at the outsourcing-prom and with the economy in shambles I’m certain it won’t be my last. I have yet to work on a project where the supposed benefits of outsourcing have actually materialized. I did some research (typed something into Google) to attempt to find out how many outsourced projects fail and saw figures between 25-40%. These seem fairly reasonable to me, and with some statistical hand-waving I will use these as “good enough” figures for the rest of this post. This leads to an interesting question though, if this strategy fails so often why do companies continue to outsource?

I would like to posit that a major reason for the continued use of outsourcing is the gap between subjective and objective measure. Subjective measures come from within, they are personal opinions, things like “Frank is a great worker!” or “Jim is so lazy” are subjective measures. Objective measures are fact-based measures things like “Frank worked 48 hours this week” or “Jim only produced 1 class file this week.”

Often, like in the examples above, our objective measures inform our subjective measures. The problem with software development is that there are no good objective measures. As much as we like to think of what we do as a science or even an industry, it is still a craft. In the above scenario we are led to believe that Frank is a “great worker!” because he worked 48 hours this week and that Jim is “so lazy” because he only produced 1 class file. If Frank worked 48 hours doing simple things and just racked up the billable hours is he really “working” harder than Jim who wrote a full web server in one week (monolithic in design so it was only one class). Getting to the heart of the value that someone produces is a difficult process and is by its nature subjective.

Here in lies the problem, software development can really only be measured subjectively, which is hard. Objective measures are much easier though, and so those are often used as a proxy for doing the difficult work of subjective measures. There have been many attempts to make sure people are performing up to par, lines of code, commits, bugs, hours worked, etc. None of these are really ideal, together they can begin to form a picture of the subjective nature of the situation, but they are not nearly enough.

Objective measures are also very seductive because you can perform calculations on them, they are easy to compare, they are easy to play out what if scenarios with. No one balks at answering the question which child is taller, but ask them which one they love more and watch them squirm. Objective measures are great because you can throw them in excel and analyze them, this is also a weakness. It’s easy to draw up a spreadsheet with your in-house assets pitted against outsourced assets and see that you can save a bajillion dollars by outsourcing, but that misses the bigger picture. What are the subjective downsides?

These subjective downsides end up failing some 40% of outsourced projects, communication, quality control, and responsiveness are hard things to chart and graph, but can easily cause a project to run off the rails. What is needed is either some way to make subjective measure easier or some fantastic new unit of objective measure. I would propose ihumanables per hour be the unit if anyone can come up with some really nifty objective system, but I don’t see that happening anytime soon, so all that’s left is making subjective measure easier, a tall order indeed.

At the heart of making subjective measure easier is really what a lot of the agile methodology is about, stand up meetings, scrums, kanban boards, they are all meant to produce a tight feedback loop so that the subjective nature of the situation is constantly being polled. I’m not going to sit here and tell you all to start running agile projects, although they can be a good bit of fun, but if you are managing people you need to get in the dirt with them. Progress reports and weekly status meetings are meant to supplement not substitute actually working with and understanding what the people under you are doing.

Remember the next time you are making a decision based off of objective measure, if you are “going by the numbers” that there are very real subjective side-effects that you should account for and be aware of. And if someone develops a really killer objective measure for software development, remember ihumanables per hour ;)

Jan 10

codemash review

I miss you

Colonel Whiskers and I have both missed you dearly

You may have noticed that I have not posted anything since last Tuesday, gasp! Fear not, although various shadow organizations are still after me, I was not captured. I had the awesome opportunity to go to CodeMash, and spent most of last week in the beautiful Kalahari Resort in Sandusky, Ohio.

CodeMash, for those of you not in the know, is a pretty cool event. Instead of focusing on one technology (.Net or Ruby on Rails for instance), there are tons of sessions spanning the technology spectrum. Want to learn about ruby, there is something for you, F#, got it covered, Clojure, for goodness sake they even did a lisp. The great thing about this style of conference is that you can pick and choose whatever interests you, nibbling on concepts here and there to get a bit of everything, or really chowing down on one topic.

The other cool thing is that the company I work for HMB is forward thinking enough to not only sponsor CodeMash, but to send me there for free! Then the big topper came last month when they asked me to present on prosper.

Well now it is all over and I can give you a rundown of how things went.

The Good

  • Kalahari – The Resort is beautiful, the rooms are huge, the staff is excellent. It is a really cool place to spend some time in, they served us some great food, and the facilities were excellent in their range, rooms small enough for a handful of people to collaborate in, all the way up to grand dining halls to house the 800 or so attendees.
  • Keynotes – Mary Poppendieck gave a great talk on Lean Software Development, Hank Janssen talked about how Microsoft is moving into the Open Source world, and Andy Hunt, co-founder of Pragmatic Bookshelf and author of the Pragmatic Programmer, gave an amusing talk on the Mother of all Bugs, our brains.
  • Prosper Talk – After weeks of preparing and fretting over my presentation I think it went well, there were some technical difficulties, but nothing that could stop the incredible power of Prosper!
  • Session Leaders – The people leading the sessions don’t get paid, they do it out of love of software development, and it shows.

The Bad

  • Superficiality – Sessions are only an hour long, this means that no matter how great the sessions are they can’t really get into the guts of anything in any kind of depth.
  • Beginner Overlap – I went to a lot of Ruby sessions, as that is my current love, and even as a n00b it became tiring to hear the 5th explanation of duck-typing for the day.
  • Kalahari is way too big – I’m just kidding about this one, but it was honestly about a 2 mile hike from the convention floor to my room.

What’s Hot

  • Ruby – Ruby is blowing up, not just rails anymore. Rails did a great job of introducing the world to the quirky little language that could. Now innovations are being made inside the Ruby community and people are starting to port that out to the rest of the software world. (See the next point)
  • TDD and BDD – Test Driven Development and even moreso Behavior Driven Development are becoming huge and are here to stay for the Agile practitioners among us. A big driving force for BDD is Cucumber.
  • Cloud and NoSQL – I’m lumping these two together because they are both technologies at scale. Virtual infrastructure is becoming a big business, be it Amazon’s EC2, Microsoft’s Azure, or Rail’s Heroku. Applications running at huge scales are also turning to NoSQL storage solutions.

The Point

CodeMash is a chance to meet really cool people how are excited about technology. It’s a chance to get exposed to a ton of cool technology. It is a chance to try a bunch of cool stuff and see what you like. Sessions are only an hour long so at worst you end up wasting 60 minutes, at best you get fired up on something wonderful. You get to meet a lot of cool people, here some great talks, and best of all you are at a gorgeous indoor water park that is 84 degrees when its freezing out in the harsh wasteland of Ohio.

Jan 10

who cares?

man shrugging

I shot who in the what now?

It’s one of my favorite questions to ask during a technical discussion. Who cares? Some others I like to throw out our, why should I care? why does it matter? and I don’t care! Am I some sort of raging lunatic completely out of my mind and trying to pick a fight with everyone I meet? Yes, but that’s not why I ask these questions. I ask these questions because as great as the big picture is we sometimes have to drop complexities to get stuff done.

When you have a big flow diagram or giant UML Class Hierarchy it can be fun to think about the abstract way that things interact and how you can pass state around and whatnot. This definitely is useful for getting a feel for a new system, but then there comes the worst part of any software developer’s job, actually getting things done. You see that’s what we get paid for, actually putting our fingers on keys, typing stuff out, compiling it, and shipping a product. Part of this is doing the upfront design work and getting all of our t’s crossed and i’s dotted, but just as important is actually getting something working.

I talked about this before in my post about simplifying things. This simple question, “who cares?” (you can be more diplomatic about it if you like) is a great tool for getting to the heart of a problem. I like the simplicity and straight-forwardness of the question, it normally catches people off guard and challenges their assumptions.

Frank: So we need to pass this data into the layer
Bob: Who cares?
Frank (taken aback): I care, I care deeply about this data
Bob: But why do you care about it being in that layer, its just going to hand it off to this layer, and that layer doesn’t care at all about it
Frank: I guess you are right, we don’t need it there

BAM!!! Problem simplified! For anyone that’s ever taken any education classes you may recognize this as a rather blunt application of Socratic Method. The point is confrontation, but not between the two people discussing something, between each person and their preconceived notions about a problem or situation.

In High School I took a chemistry class, it is one of the few classes that I really remember in any detail from so long ago, the teacher taught the entire semester in Socratic Method. He would question us, we would experiment, we would reason out why we observed what we observed, he prodded us, and we cherished our knowledge. Anyone can tell you water is composed of 2 hydrogen atoms and 1 oxygen atom, but when you had to work for it, when you had to defend your knowledge of it from scrutiny, it became concrete, it became your discovery.

Applying this to software development is useful, we can get locked into our preconceived notions, and direct, blunt, sometimes even rude questions can spur you to really consider what you are talking about. Software is incredibly fluid, sometimes an idea or concept can become out-dated in a matter of months or a matter of hours. The biggest ones are the most difficult to challenge, because they often have the most dependent code around them, but this makes them the most important to constantly consider.

Keeping your conceptual design, implementation, and solution in lock step with a problem domain often involves some amount of change, that change can be hard to come by though. By challenging our standing beliefs we can find new avenues for exploration or strengthen the existing techniques and gain a better understanding of them.

Jan 10


monkey using rock as a hammer

Pictured Above: Programming

For about a year I had the unique experience of working at Abercrombie & Fitch’s world headquarters in New Albany, Ohio as a Java Developer. It was a really interesting place to work, very stylish, loud music all the time, complex problems of incredible scale, and some of the best developers I’ve ever worked with. I was lucky enough to get placed into one of the most successful development teams, Planning. This team had developed a tool that was hugely popular and well regarded, and I was coming on board to maintain and improve this tool. The beautiful people and board-short dress code was just icing on the cake.

As a software developer this is a fairly basic scenario, there is some task that is complicated, business users want it to be less complicated, they give me some money, I make them some program, and everyone is happy. The odd thing though is that while programmers are completely used to making tools for others, they often forget that they can leverage this amazing skill for their own benefit. I distinctly remember a conversation I had with one of my colleagues there, Steve. It centered around the idea that most developers are happy to just get their assigned tasks done, but there are some that are toolmakers, and being a toolmaker is more valuable to a company than being a developer.

Fast forward to after my stint at A&F, I was on a new project and given the task of translating 80,000 lines of PL/SQL into T-SQL. The idea was to use…um…find and replace…or….something…and…um…get this to work. With such a fine technical specification and a mountain of PL/SQL staring at me, Steve’s words bubbled up to my brain, “Make a tool.” I had a time window of a some number of weeks to do the translation, it was a bit of a gamble, but I decided to spend a week writing a tool.

After remembering how C# worked after a year in the trenches with Java I had a program that took in the PL/SQL parsed each line and translated it to T-SQL. The task was made easier by the fact that I didn’t have to write a generalized parser for PL/SQL as the 80,000 lines were written in a fairly consistent style, which greatly simpified the translation tool. After running the tool a few times, fixing some bugs, I had finished the task in a fraction of the projected time. Triumphantly, I went to my PM to show how I had just saved weeks of manual laborious work. “Great, now here is your next task….”

ticker tape parad

Expected Reaction

guy giving an overenthusiastic thumbs up

Actual Reaction

So I didn’t get the praise I was expecting, that’s fine, I’m not 4, I don’t need a cookie. I continued on in my tasks quietly pleased with saving myself weeks of tedious work. Then the day came when the database requirements changed, the hundred or so CREATE TABLE statements would need to be altered. The task wheel was spun, and since only my name was on it at the time (thus defeating the need for the wheel) I was the lucky person to go through and change various aspects of each table.

I thought to myself, “Great, there goes a day or two down the grunt-work hole.” Then the part of my brain that I put in charge of remembering important things woke up and slapped me. “You haven’t altered the SQL yet, just alter your tool to do this too and re-translate, dumbass.” After pondering the suggestion, and possibly more importantly the insult, leveled at me by my brain, I opened up my translation tool. A few keystrokes later the change to the translation mappings was made and I fired off the program. Bits flew, gears cranked, and out came the exact result I was looking for, the tool had just saved me another day or two.

Ultimately the project was a big hit, we delivered on-time and on-budget for what was a somewhat aggressive schedule. The tool saved us a good amount of time, let us get the application bootstrapped and running, which allowed us to start iterating and get things working. Over the course of that project I created tools here or there that ended up saving me, and some thankful members of my team, time and effort.

The lesson here is that you should look around you and see if you can make some tools. Tools can be scary though, they are a bit of an upfront cost, and if it doesn’t work out then you’ve wasted even more time. With time and experience it becomes easier to recognize when it’s a good time to make a tool and when it’s a waste. Here are some good guidelines for tool making success.

  • The best candidates are tasks with high amounts of repetition and little logic.
  • Check to see if a similar tool already exists, don’t reinvent the wheel.
  • YAGNI (You Ain’t Gonna Need It). You should be intimately familiar with the problem you are going to solve, solve exactly that problem, you can scale it out later. I didn’t bother writing a general purpose PL/SQL to T-SQL translator, I only had one PL/SQL file to translate, if it worked on that then it worked on 100% of its input.
  • If it would take less time to do the boring thing manually than to build a tool, just suck it up and do the boring manual task
  • Share your tools, even if you don’t think they will be useful to anyone else on your team, sometimes a tool can provide a surprising benefit to others
  • KISS (Keep It Simple, Stupid). This is no time to become a highfalutin architect. Tools are normally quick and dirty one-offs, normal concerns like efficiency, user-friendliness, etc. can be brushed aside. If the tool is getting used often and by many people you can refactor it to make it more efficient or user-friendly, but don’t pay that cost upfront

Developing software often requires a high degree of focus on the problem at hand, I wrote the other day about one approach to problem solving, simplifying the problem. This strategy can be coupled with toolmaking to dramatic effect. Once you realize the simpler actions you can automate those and write a tool to perform the task for you. The next time you are facing some mountainous problem, consider the toolmaking route.

Nov 09

division of labor

cartoon labor

cartoon labor

When building or analyzing any system of significant complexity the most important thing to determine is a proper division of labor. The entire idea of n-tier architecture is built on dividing labor up into the appropriate tiers and keeping them cleanly separated. For anyone that’s ever worked on a real world system half of the difficulty is caused by one layer leaking into another and the other half is getting around the separation caused by separation.

Despite my last sentence, division of labor is a necessary and good part of design. When done right it allows you to know where functions, variables, and constants should live, how things should interact, and who is the system of record. This is why it is imperative to decide beforehand or shortly into a project what the lines of demarcation are and who is responsible for what.

Prosper is one of a group of libraries, with well defined divisions of labor. This allows the libraries to be decoupled, and also has the added benefit of letting me know when I’m done. When I found myself adding functionality to Prosper the other day that I knew belongs in the next library, I realized that it was time to officially start the next library. So a new skunkworks project is moving ahead full steam, it contains 2 parts that work together, and so again requires a well defined division of labor.

Just because work has begun on a new library doesn’t mean that Prosper is left out in the cold, yesterday I added support for the deprecated mysql library, standardized construction, and added better support for native functionality. If you are checking out Prosper, you can follow the latest development by pulling from the svn repo.

Back on topic though, figuring out the division of labor is the most important part of any analysis or design work. We recently at work had a 2 hour long discussion about order numbers. It went something along the lines of this:

BA: This order number comes from the Mainframe
Expert: No the Application creates that number
Dev: So, what does the SOA layer need to do?
BA: Generate the number
Expert: No, do nothing, the Application will automatically do it
Dev: Wait so who is responsible for creating the order number, and if its not the Mainframe why is it making one?
Expert: The Mainframe always made that number and the Application has always ignored it.
BA: So we will make it in the SOA Layer?
Dev: Who is in charge of making the unique identifier order number?!
Expert: [shrug]?
BA: [shrug]?
Dev: Let’s decide that first

Explicit division of labor requires you to pull the unspoken assumptions everyone has about the system out of the silence of our own minds and puts it into the blinding daylight. Suddenly communication is smoother, arguments come to an end, because there is some record to consult to help guide decisions.

Draw out your explicit divisions of labor, don’t carve it in stone though, if your original design ends up being too inflexible, allow it to change, but again, bring all the unspoken assumptions back out into the open. This allows you to cleanly define what piece is in charge of what function, keep layers from leaking into each other, and make for smoother development and analysis.

Nov 09

clean slate

Today is my first day at my new assignment. My company gave me a new(er) corporate laptop with a plain vanilla install of Windows XP Pro to use at my client site. Last night I had the joy of looking upon that fresh computer and thinking of the wonders that could be. I got ready to install the base programs that take a windows machine from factory to awesome. Here is my essential install list.

Then I spent plenty of time monkeying with the filesystem, the wallpaper, things that don’t particularly matter, but make the laptop feel like home to me. I made my icons and text smaller, since I’m used to a giant monitor and the 1280×800 resolution felt a little cluttered to me. Now though I feel right at home with this machine, and feel like this is the best possible time of laptop ownership, right after you’ve got it set up just the way you like it, right before you add stupid things that pollute the registry and mess up your beautiful filesystem.

The other clean slate, the less fun one, is my brain. I have all kinds of knowledge in my brain about my last projects, but today I find myself swimming in a see of paths and acronyms and weird buildings, and a chair that isn’t quite high enough to be comfortable. I’m starting to wrap my mind around the acronyms and getting a grip on what exactly we are doing here, but I fear it will take more time to feel as comfortable with this new position as I do with my new laptop.

Suffice it to say it could be much worse, I can clearly still push to my blog, so that’s sweet. The people I am working with seem cool and fun, the project seems like a lot of work but nothing that we can’t handle. I will probably have a whole different viewpoint in a week when I truly realize the scope of the project and settle into a nice little routine.

So far, so good, I’m off to read another IBM pdf that is half marketing / half documentation. Wish me luck.