Today I conquer steps, tomorrow France!

Last night I had the unadulterated fun of updating a client’s blog. After handing off the reigns everything was going smoothly until some unknown combination of plug-ins was causing the mobile version of the site to be served to actual browsers. I was called in to triage the situation, nothing too dire, just two plug-ins with some configuration problems, once I told them to play nice, all was fine. It was then that I noticed there were 15 plug-in updates waiting and the whole WordPress install needed to be updated. So I took the time to take a backup and update everything, nothing too serious (for anyone that doesn’t use WordPress, updating is a one-click affair now) just the day to day stuff that needs to be done to keep the lights on.

This got me thinking about upgrading and Google’s decision to discontinue support for IE6. Some people are crying foul, saying that IE6 should still be supported, I’m decidedly of the I-hope-IE6-dies-in-a-fire camp. Lest this become a let’s bash on IE6 rant, which it so easily could, fucking broken box-model CSS1 piece of shit, let’s get back on track. The people that are upset about this are not technical luddites clinging to the familiar. Digg performed a user survey to figure out what they should do about IE6 and part of that was asking why people used IE6. Of IE6 users 70% responded that they either lacked administrative rights needed to upgrade IE6 or were told by someone at work that they could not upgrade. For anyone that has ever worked in Corporate America this should be no surprise to you. The nagging question though is why?

Why would the IT department, home of nerds always hell bent on trying out the latest and greatest, demand that you use a 9 year old piece of software? There are a number of reasons people point to.

1. Mission Critical application is only compatible with IE6
2. IE6 comes installed on all the machines
3. IE6 is maintained by Microsoft, less of a headache for us
4. Licensing / Partnership / Legal-mumbo-jumbo

If you fall into #4, I pity you. #3 is laughable because although the updates might be bundled with Windows Updates you aren’t saving yourself any administrative costs by putting the most targeted web browser for malware on all your machines. #2 is true, but if you are big enough for an IT department you probably have some way to push software to machines (or you could, and this is a shocking idea, trust your employees to choose a browser that fits their needs). I really, really, really want to talk about #1 though, because that is the point of this whole post.

I’ve worked at a number of places that used some sort of Web Application that was deemed critical that only worked right in IE6. This was then used as a justification for never moving past IE6, the timecard application won’t work or the spline flanger won’t have those cool IE6 exclusive filters. I understand that no one wants to expend resources on a “working” application pulling it into the future. But what is the longterm idea here, are you just going to use IE6 forever? This prevailing consensus seems to be ultimately self-defeating. Google recognizes this and as they try to push what can be done with the web they find themselves expending far too much time and energy on a nearly decade old piece of technology.

The nice part about fixing an application on IE6 is that you avoid some cost in updating it to make it work with other browsers like IE7. You could have spent a week or two adding in some CSS fixes and some markup changes and bingo bango IE7 now works like a charm. Instead you decide to use Corporate Fiat to force the continued use of IE6. Well now IE9 is looming around the corner and instead of paying a week here or a week there you now have a mountain of technical debt to pay down to get it to work.

Although IE6 is a convenient example, we can see this happen in all parts of the software development world. This program compiles when linked to an older version of the library but not this new one, I don’t have the time to figure it out, well just link in the old one. This is all well and fine to meet the deadline but you have just accrued some technical debt. Later when that library gets some great new speed-up or killer new feature, there you will be plodding along with version 0.91 because you can’t get it to work with the new hawtness. The problem is that these debts are not additive, they compound on one another and make what would be a simple conversion to 0.92 a nearly impossible conversion to 3.76.

Keep your tools up to date, keep an eye on what’s up and coming, you don’t have to be the first to jump on the latest and greatest but when a new version has proven itself and become the new standard move to it. Technology doesn’t stand still, if you don’t keep up with the latest stable releases you will find yourself in a tight spot trying to do an overnight re-engineer of your product. Take the easy baby steps from version to version so you don’t find yourself having to take a technological leap down the road.

Pictured Above: Programming

For about a year I had the unique experience of working at Abercrombie & Fitch’s world headquarters in New Albany, Ohio as a Java Developer. It was a really interesting place to work, very stylish, loud music all the time, complex problems of incredible scale, and some of the best developers I’ve ever worked with. I was lucky enough to get placed into one of the most successful development teams, Planning. This team had developed a tool that was hugely popular and well regarded, and I was coming on board to maintain and improve this tool. The beautiful people and board-short dress code was just icing on the cake.

As a software developer this is a fairly basic scenario, there is some task that is complicated, business users want it to be less complicated, they give me some money, I make them some program, and everyone is happy. The odd thing though is that while programmers are completely used to making tools for others, they often forget that they can leverage this amazing skill for their own benefit. I distinctly remember a conversation I had with one of my colleagues there, Steve. It centered around the idea that most developers are happy to just get their assigned tasks done, but there are some that are toolmakers, and being a toolmaker is more valuable to a company than being a developer.

Fast forward to after my stint at A&F, I was on a new project and given the task of translating 80,000 lines of PL/SQL into T-SQL. The idea was to use…um…find and replace…or….something…and…um…get this to work. With such a fine technical specification and a mountain of PL/SQL staring at me, Steve’s words bubbled up to my brain, “Make a tool.” I had a time window of a some number of weeks to do the translation, it was a bit of a gamble, but I decided to spend a week writing a tool.

After remembering how C# worked after a year in the trenches with Java I had a program that took in the PL/SQL parsed each line and translated it to T-SQL. The task was made easier by the fact that I didn’t have to write a generalized parser for PL/SQL as the 80,000 lines were written in a fairly consistent style, which greatly simpified the translation tool. After running the tool a few times, fixing some bugs, I had finished the task in a fraction of the projected time. Triumphantly, I went to my PM to show how I had just saved weeks of manual laborious work. “Great, now here is your next task….”

Expected Reaction

Actual Reaction

So I didn’t get the praise I was expecting, that’s fine, I’m not 4, I don’t need a cookie. I continued on in my tasks quietly pleased with saving myself weeks of tedious work. Then the day came when the database requirements changed, the hundred or so CREATE TABLE statements would need to be altered. The task wheel was spun, and since only my name was on it at the time (thus defeating the need for the wheel) I was the lucky person to go through and change various aspects of each table.

I thought to myself, “Great, there goes a day or two down the grunt-work hole.” Then the part of my brain that I put in charge of remembering important things woke up and slapped me. “You haven’t altered the SQL yet, just alter your tool to do this too and re-translate, dumbass.” After pondering the suggestion, and possibly more importantly the insult, leveled at me by my brain, I opened up my translation tool. A few keystrokes later the change to the translation mappings was made and I fired off the program. Bits flew, gears cranked, and out came the exact result I was looking for, the tool had just saved me another day or two.

Ultimately the project was a big hit, we delivered on-time and on-budget for what was a somewhat aggressive schedule. The tool saved us a good amount of time, let us get the application bootstrapped and running, which allowed us to start iterating and get things working. Over the course of that project I created tools here or there that ended up saving me, and some thankful members of my team, time and effort.

The lesson here is that you should look around you and see if you can make some tools. Tools can be scary though, they are a bit of an upfront cost, and if it doesn’t work out then you’ve wasted even more time. With time and experience it becomes easier to recognize when it’s a good time to make a tool and when it’s a waste. Here are some good guidelines for tool making success.

• The best candidates are tasks with high amounts of repetition and little logic.
• Check to see if a similar tool already exists, don’t reinvent the wheel.
• YAGNI (You Ain’t Gonna Need It). You should be intimately familiar with the problem you are going to solve, solve exactly that problem, you can scale it out later. I didn’t bother writing a general purpose PL/SQL to T-SQL translator, I only had one PL/SQL file to translate, if it worked on that then it worked on 100% of its input.
• If it would take less time to do the boring thing manually than to build a tool, just suck it up and do the boring manual task
• Share your tools, even if you don’t think they will be useful to anyone else on your team, sometimes a tool can provide a surprising benefit to others
• KISS (Keep It Simple, Stupid). This is no time to become a highfalutin architect. Tools are normally quick and dirty one-offs, normal concerns like efficiency, user-friendliness, etc. can be brushed aside. If the tool is getting used often and by many people you can refactor it to make it more efficient or user-friendly, but don’t pay that cost upfront

Developing software often requires a high degree of focus on the problem at hand, I wrote the other day about one approach to problem solving, simplifying the problem. This strategy can be coupled with toolmaking to dramatic effect. Once you realize the simpler actions you can automate those and write a tool to perform the task for you. The next time you are facing some mountainous problem, consider the toolmaking route.

Just like this, only less beautiful women and more boring data

When solving a problem the most difficult part should almost never be the implementation. Implementing a solution should be fairly straightforward from the solution itself, if it is not, then your solution is incomplete. That doesn’t mean that you need to sit down with pen and paper and completely solve a problem before coding, this is an approach taken by some books attempting to teach software development. Some go as far to advocate for an even stricter approach, something along these lines.

1. Model the problem domain with UML or some other modeling technique
2. Translate your modeling into pseudo-code
3. Translate pseudo-code into actual code
4. Realize 2 month project took 2 years to complete
5. Go to step 1 (the problem has radically changed since you took so long)

Of course, I’m poking fun at the books that take such a structured route. Here is a news flash for everyone Programmers are lazy. Here is another news flash (and this might actually be news to some people) Being lazy is good. Not all laziness is good, if it causes you to cut corners and fail to deliver a high quality product, than you fail at life. If however your laziness drives you to find an easier way to do things then “Lazy FTW!”

I have the “joy” of programming Java in my 9 to 5, when you write Java for a living you get used to the absurd amount of verbosity in your code, you also have the joy of Eclipse (notice no quotes around joy) that will write most of your Java code for you. Then there are things like Project Lombok that strive to make Java easier and more fun to write. C# got the memo, and let their programmers be lazy, let’s take a look at attributes in a few languages

Here is some C# code to make a class called Foo with a string member called Bar that can be read and written

public class Foo {
  public string Bar { get; set; }
}

They realized people are lazy, let’s look at the same thing in ruby

class Foo
  attr_accessor :bar
end

Again, concise and simple, let’s look at some Java

public class Foo {
  private string bar;

  public string getBar() {
    return this.bar;
  }

  public void setBar(string bar) {
    this.bar = bar;
  }
}

Can you hear it? I can, it’s the language architect saying “Just don’t be so lazy, write those simple functions, it isn’t hard.” The truth of the matter is that it isn’t hard, in fact Eclipse will happily write these functions for you, just hit CTRL-SHIFT-G. I’m sure there is a point here that I’m getting to, and here it is. I don’t want to disparage Java or start a language war, what I want to point out is that Java has a different conceptual model than C# or ruby. C# has the concept of properties and ruby has the concept of attributes, Java doesn’t have anything like this, classes have members and functions, nothing more.

The point is that the conceptual models your solution adopts will have huge ramifications on the implementation of the solution. There are many examples of unifying philosophies that make things simpler. Unix’s concept that everything is a file, C++ concept of streams, Erlang’s pervasive share-nothing message passing, ruby’s duck typing. These are core concepts that have profound and far reaching consequences.

Working on a new side project I recently was struck by a way to simplify a complex data structure by adopting a new conceptual model. Creating an internally consistent, elegant conceptual model is the most important thing you can do for a project’s success.

Conceptual modeling is a hard thing to get right, go into too much detail and your model becomes rigid and brittle, go into too little and the model can be impossible to implement correctly. Making a good conceptual model is like making a good analogy, there are a few hallmarks to a good conceptual model.

• The conceptual model should simplify the problem domain. Files and folders greatly simplifies the concept of a hierarchical file system by relating it to something commonplace.
• The conceptual model should not be overly complex to translate into an implementation
• The conceptual model should be uniform and consistent, if you find parts of the problem domain falling outside the conceptual model or feeling “tacked on” you should rework the model

The next time a project seems stuck or overly complex, instead of refactoring code or attempting to throw another library at it, take a step back, look at the big picture and refactor your conceptual mode.

you can find really cool images searching for impedance mismatch, this one has lazers (with a z)

There is a threat to good software, one that threatens the most promising of software projects. This threat is impedance mismatch. I use impedence mismatch in the sense that Object-relational impedance mismatch uses the term, but make no mistake, I’m not talking about the difficulties of writing an ORM. What I wish to discuss is far more insidious, it is death by a thousand paper cuts, it can spell ruin for any project. Impedance mismatch is born out of the confusing first steps of a project, it can also arise from multiple developers working over time on a large project or from internal disagreements over the optimal solution.

The major problem is that when we go about designing some new piece of software, some whiz-bang solution, we do not fully understand the problem domain. Any good development team will spend days filling up whiteboards with the solution that models the problem domain, but anyone with experience will tell you that once you get into the guts of a problem you realize that things need to be changed. This is fine, in fact this is a sign that you are solving the actual problem and not the ideal one you began with, don’t worry about changing your solution to meet the realities of the domain. The problem though is making these changes first class citizens and not some weird third class monster thrown into the mix.

Have you ever been on a project and heard a conversation like this?

New Developer: What is X?
Grizzled Veteran: X is what we refer to internally as Y
New Developer: Why do we call it Y internally but X on the UI
Grizzled Veteran: Time constraints… budgets…. other misc. bullshit
New Developer: *mental note: X is called Y sometimes*

BAMMO! There’s some nasty impedance mismatch there. The problem is not so much in what we display to the user, or what our object properties are called, or what the database columns are named, the issue is that we have made our mental model convoluted. At the heart of all these systems we build as software developers is some sort of mental model, at runtime this is expressed in system as an object graph, for persistence it is expressed as a collection of tables and relationships, but all of these things are just a representation of some mental model.

artist depiction: working with an elegant mental model

When our mental model is internally consistent and simple and elegant, it makes life easy and happy and unicorns shoot out of our monitors and everything is sunshine and puppies. When your mental model gets convoluted it can be an awful and bewildering experience. The problem comes from an increased overhead, instead of the mapping from one representation to the next being clear-cut, the developer has to carry around a hashmap in their skull mapping various representations of their mental model. The difficulty of carrying out even trivial tasks is increased as the mapping from X -> Y -> Z -> Q -> P takes place.

impedance mismatch vs. coupling: battle to the death

So let’s just name everything the same and go home, problem solved. Oh, if it could only be so easy, the problem is that on the other end from impedance mismatch is a devilish little troll named coupling. Coupling is the amount that two pieces of software are intertwined, they could be tightly-coupled wherein changing one requires changing the other, or they can be loosely-coupled all the way to the point of being considered decoupled or orthogonal. Orthogonal pieces are nice to work with because you can change one and know that you won’t have side-effects on other pieces. Orthogonality goes by some other names: encapsulation, modularity, and decoupled, it is generally considered a Good Thing© and I make no argument against it.

We now have these two seemingly opposing forces, we want to decrease impedance mismatch without introducing coupling, what should we do? There are many ways to deal with this issue, let’s look at a few.

Adapters are nice little pieces of code (also called wrappers), that allow two different things to look the same. Let’s assume we want to write some code to manage users, users have a name, an address, and an age. The developer of the class decides that this object can be pretty simple.

class User {
  public $name;
  public $address;
  public $age;
}

The database developer decides that he loves structured data and that this should be broken out into all kinds of columns and tables.

table user (
  user_id int autoincrement primary key,
  first_name varchar,
  last_name varchar,
  addr_street varchar,
  addr_number int,
  addr_city int foreign key,
  addr_zipcode int,
  age int
)

table city (
  city_id int autoincrement primary key,
  city_name varchar,
  state_id int foreign key,
)

table state (
  state_id int autoincrement primary key,
  state_name varchar,
  state_abbrev varchar
)

We could create an adapter class that bridges the difference, for brevity’s sake I won’t include the code for it, but it would basically translate strings into database fields, as the application developer I get to imagine the database as mirroring my object, even though it doesn’t.

ORM – ORM’s are like automatic adapters. They are not without their faults and some implementations are better than others, but they mask the complexity of marshaling data and reduce the impedance mismatch.

Convention over Configuration – This was made famous by Ruby on Rails. There is some strong convention that you follow that allows for a consistent mapping between domains, an example would be that tables are always named the pluralized version of the class name.

There is no silver bullet to solving the problem of impedance mismatch, but a combination of the above can help limit it. The best thing you can do is be aware of the problem, don’t introduce impedance where it is unnecessary, and never be afraid to ruthlessly refactor out impedance mismatch when you find it. Keeping impedance mismatch low increases developer productivity, lowers the bar for bringing on new people, and makes unicorns leap out of your monitor farting rainbows.

don't worry, no babies were harmed

Yesterday my friend Ian sent me a tweet about prosper

@ihumanable I’d be more interested in using Prosper if it provided support for true prepared statements instead of concatenating strings.
6:29 AM Dec 7th – pian0

With that my baby was born, the baby was a version of prosper that utilized prepared statements where appropriate. Prosper’s main functionality has been coded for a while and the bulk of the work I’ve been doing as of late has been testing, improving backend support, and adding new functionality. The nice thing about this kind of work is that you don’t have to go mucking around in working code, you are just fixing broken stuff and adding new functionality. The idea of adding prepared statements made me a little queasy, I had briefly looked at it when I was starting prosper but decided that they were so varied that for the first attempt I would just avoid the headache. What’s the problem you ask, let’s take a look at this feature table real quick.

As you can see the support for prepared statements varies in the Vendor Specific Database Extensions of php. There is also the fact that prepared statement support is poorly implemented in several extensions, they can be coded around but it adds a good deal of complexity. Despite all the reasons not to implement prepared statements, safety, speed, and correctness dictate that I should at least give it a go.

Yesterday and today were my go, and I met with some success. Prepared statements are not all that difficult to implement for a given backend, throw in some question marks, call the correct bind function, and you’re off and running. The real difficulty is the cross-platform nature of prosper. I wanted prepared statements to be a first class citizen, I didn’t want to decrease any existing functionality, I wanted to not overly complicate the way prosper works, and because there are 11 adapters that need to be upgraded, I wanted it to be simple to add this functionality to a given adapter.

I worked and toiled and toiled and worked, I tried hacking something up to make the MySQL adapter work, ran into the bind_results function and screamed at it for a while. Then I got out some paper and drew out some diagrams, thought about sample code. Then I went to sleep. Then I woke up and in the shower pondered the proper syntax for passing around typing information and the division of labor for transformation. Then I hacked on it some more and wrote a quick test. Then and this is the most important part I realized I had done it absolutely backwards.

I had some clues that stuff wasn’t working out right, I hit a point where I would either need to embed an if statement in every function of an adapter or have a duplicate function. I thought that with some clever coding I could shove this down into the base class, it wasn’t great but it was working. Then I saw another red flag when I realized that the processing in the where clause was identical to the processing in the values clause, and I applied some refactoring. This is when I realized that there was a single function that I could have modified and had the whole thing work much more elegantly.

That is when I had to come to the decision to murder my baby. I had painstakingly worked out the kinks for two days, traveling far down this road, touching a bunch of code, and now I realize the best way forward is back. The best thing I can do right now is to revert my changes to Monday morning and lose 2 days of work, and that is great!

It’s not the best outcome, in an ideal world I would have seen this solution before and applied it and not wasted 2 days. The thing is that I’ve learned a great deal, I’ve come up with a much better implementation, and I’ve been reminded of an important lesson, never be afraid to throw away code. The code I wrote is adequate, it works well enough, but it is far from optimal. I could keep going, head down, plowing away making as many problems as I solve trying to build a house on an unsound foundation. The point here is to pay attention to those red flags.

The other important point is that sometimes you have to actually try something to find the thing that will work. You may lose some time, but even after sitting down with pencil and paper, thinking long and hard, the optimal path didn’t come to me until I walked down the wrong road. That’s just life sometimes as a software developer, you have to learn to live with it.

I will continue working on prepared statements for prosper, and I hope that by 0.7 or 0.8 they will be implemented and in there. I still question the syntax and think that I can clean it up, right now the plan is for prosper to support unnamed parameters (?), named parameters (:name), unnamed typed parameters (%i), and named typed parameters (:name%s), but all this seems like a little much. Good thing prosper hasn’t had a 1.0 release yet

• First Dev: I want this new project to seamlessly integrate with eBay, digg, and amazon
• Second Dev: Those seem arbitrary, maybe we should define some sort of data import protocol
• First Dev: Yea, then we just need a scraper piece that uses the adapter pattern to translate various websites into the standard import protocol and feed the project
• Second Dev: I’ll start defining a dtd and you can start writing up some xslt for eBay and digg and amazon as a proof-of-concept
• First Dev: Great, then we just need to get the exchange support and the poor man’s cron jobs running, to send out the periodic updates
• Second Dev: Should we integrate with Twitter?
• First Dev: I don’t see why not
• Third Dev (interrupting): Wait, what are we developing?
• First Dev: Todo list
• Third Dev: Maybe we should get it to store and delete todo list items first
• Second Dev: That’s the easy part, the multitouch integration is going to be the hard part
• Third Dev: …

the dream

Before we get the product to do something simple, something any moron could code up, we want to start slapping on features and bells and whistles and whizbangs. This is all fun, because whizbangs are fun things to think about and code and its all well and good until you realize you have coded up a great feed reader, but one hell of a lousy todo list. Focusing on least viable product in the beginning gives you a concrete base to start applying whizbangs, based off of real world feedback and grounded in reality. Incrementally building up value off of an existing product is easier and makes more sense than building a giant all-in-one application that can do everything but butter your toast that no one wants to use.

So this leads me to the best way forward in my estimation and experience, nothing. Nothing is incredibly powerful, tremendously liberating, and allows you to get something up and running in least viability mode or to solve a complex issue.

I can hear it already, what the hell are you talking about you mentally unstable person?! Let’s look at some tools that exemplify nothing and how you can go from nothing to something quickly and incrementally, but only if you start with nothing.

Mock-ups. What is a mock-up, nothing really, just a fancy picture of what you want an interface to look like. There are many great tools for making mock-ups, a new one that just hit beta is mockingbird, a more seasoned veteran in this game is the excellent Balsamiq. These shouldn’t help, we all know what a todo-list looks like, why would mocking one up help us? Because, even though we are capable of abstract thinking it is much easier if we have a concrete model to wrap our heads around. Like I said in tool roundup

[talking about a sweet visualization I made] …but it is a much better reference when working with this particular part of the system than looking at the database table and trying to assemble parts of it in my brain as the situation demands.

If we don’t have a concrete model we force our brains to construct one on the fly, taking us out of our work and into a conceptualization, having a concrete model frees up precious processor cycles in our brain. A mock-up is also great because it allows us to take nothing, a simple functionless mockup, and quickly turn it into an actual interface. What does that button do? Nothing! What about that one? Nothing!!! Then we can slowly but surely add functionality to our interface until we have a functional least viable product. The way we got there was by understanding that we start with nothing and work our way gradually to something. Sure we could have sat down at the keyboard and pounded away code for a day or two and then run the thing, but the faster we can get it on the screen and start iterating over it, the better.

The second way that nothing can be useful is when tackling a huge problem, lets look at the contrived feed reader problem from up above. How in the world do we go about solving that?! Well we employ are good friend nothing, and we build from there. What’s would the simplest possible solution be?

$feed = FeedReader::read("http://www.digg.com");

We can throw that in our program, now we need to deal with the FeedReader class

class FeedReader {
  static function read($url) {
    return '';
  }
}

Nothing to the rescue, we just defined our super simple API, but we aren’t quite to functional yet, but we have a foothold now to iterate from.

class FeedReader {
  static $readers = array('http://www.ebay.com' => new EbayReader(),
    'http://www.digg.com' => new DiggReader(),
    'http://www.amazon.com' => new AmazonReader)); 

  static function read($url) {
    $reader = self::get_reader($url);
    if($reader) {
      return $reader->process();
    }
  }

  static function get_reader($url) {
    return self::$readers[$url];
  }
}

Wow look we are almost all the way to functionality, amazing!!! Thank you nothing (that’s what EbayReader, DiggReader, and AmazonReader do). Now its the simple task of implementing those, which each by themselves is a somewhat complicated task and outside the scope of this blog post.

This sort of top down design though can be very helpful, you need to solve a problem, just pretend that you have an API that already solves it and start calling it, stub out those functions (stub is a fancy computer science term for “make it do nothing”), and watch the power of nothing first hand. Then you fill in the blanks, allowing yourself to have more nothing functions, you successfully break up your huge monster task into simple solvable tasks.

When we combine the first technique with the second we can get truly awesome results, we normally call this Rapid Application Development (or the much cooler 1980′s name, RAD). RAD can get us to something faster than almost anything, using the power of nothing.