Monthly Archives: May 2011

How to Read Industrial Control System Wiring Diagrams

I write a lot about the PLC side of industrial automation, but it’s also fundamental to have a good foundation in the electrical side of things.

First of all, most modern (North American) industrial control system wiring diagrams have a relatively common numbering scheme, and once you understand the scheme, it makes it fairly easy to navigate the wiring diagram (commonly called a “print set”).

Let’s start with the page and line numbering. Most multi-page wiring diagrams use a two digit page number (page 1 is “01”). In the rare, but possible, event that you end up with over 99 pages, some diagrams will just add 3 digit page numbers (starting at “100”), but if there was any forethought, many designers will divide their wiring diagrams into sections, giving each section a letter (let’s say “A” for the header material, “B” for power distribution, “C” for safety circuits, etc.). Within each section, you can re-start the page numbering at “01”. This has the added bonus of letting you insert more pages into one section without messing up the page numbering.

Within a page, you’ll typically see line numbers down the left side (and frequently continuing down the middle if you don’t need the whole width of the page for your circuits). These numbers will start with the two digit page number, followed by a two digit line number. Typically these start at zero, and increment by twos:

  • 1000
  • 1002
  • 1004
  • 1006
  • … and so on

Now, devices (like pushbuttons, power supplies, etc.) usually have a device ID based on the four digit line number where they are shown in the wiring diagram, possibly with a prefix or suffix noting the device type. So, if you have a pushbutton on line 2040, the device ID might be PB2040 or 2040PB. The device ID should also be attached to the device itself, normally with an indelible etched label (lamacoid). Therefore, if you find a device in the field, you should be able to find its location in the wiring diagram. (Finding the wiring diagram, of course, is often the more difficult task.)

Wires are numbered similarly. The wire number is typically based on the four digit line number where the wire starts, plus one extra digit or letter in case you have more than one wire number starting on the same line. So, the wire numbers for 2 wires starting on line 1004 might be 10041 and 10042.

It’s typical for wires to connect to devices that are on other pages (it’s extremely common, in fact). In that case, you’ll see off-page connectors. The shape of these vary based on whose standard was used for the wiring diagram, but they’re typically rectangles or hexagons. In either case, inside the shape will be the four digit line reference number where the wire continues. The other end of the off-page connector (on the other page) also has an off-page connector in the opposite direction. Note that you frequently see connectors from one place on a page to another place on the same page, if it happens to improve readability.

That’s all you really need to know to find devices and follow wires in a wiring diagram. Now, to understand the components in an industrial control system, that’s going to take longer than a blog post. For a great introduction, I recommend the book Industrial Motor Control by Stephen Herman. Google books has a great preview if you want to check it out.


What Motivates Makers

A couple weeks ago I wrote a post about budgets, and Ken from over at Robot Shift replied:

My experience when compensation is set up to be win-win it’s a good thing. If someone’s working hard, and doing the right things often their impact far exceeds that of just their hourly contribution. It impacts teams, groups and profitability. If the organization profits from this impact, the employee should as well.

On the surface I think that’s true – we all appreciate profit sharing, but I think Ken’s talking about performance-metric-based-pay. I don’t agree with tying creative work to performance pay. The inner workings of the human mind continue to astound us. An article in New Scientist says:

…it may come as a shock to many to learn that a large and growing body of evidence suggests that in many circumstances, paying for results can actually make people perform badly, and that the more you pay, the worse they perform.

“In many circumstances” – I’m going to talk about one such circumstance shortly. Of course for more surprising information there’s the classic Dan Pink on the surprising science of motivation, and Clay Shirky’s talk on Human Motivation.

There’s a big difference between intrinsic motivation and extrinsic motivation. If you have to choose, intrinsic motivation is more powerful. It costs less, and it’s a stronger motive force. I believe that’s due to the phenomenon of “flow“. If you work in any type of creative field, you know the power of flow – that feeling you get when you’re 100% immersed in your work, the outside world seems to drift away, you lose track of time, but you’re incredibly productive. Without distractions you can focus your full attention on solving the problem at hand. Once you’ve experienced this a few times, it’s like a drug.

For someone like me, assuming I make enough money to pay the bills, I would trade any additional money to spend more time in “flow”. Yes, I’m Scott Whitlock and I’m addicted to flow. (I’m not the only one.) If you have one of those lucky days where you spend a full 8 hours without distractions, wrestling with big problems, you’ll come home mentally exhausted but exhilarated. At night you truly enjoy your family time, not because your job sucks and family time is an escape, but because you’re content. The next morning you’re excited on your drive to work. Your life has balance.

Paul Graham wrote an excellent essay about the Maker’s Schedule, Manager’s Schedule. The Maker blocks out their time in 4 hour chunks because they need to get into “flow” to do their best work. Managers, on the other hand, schedule their days in 30 or 60 minute chunks.

Do you see the problem? If a Manager never experiences flow then they can’t understand what motivates their Makers. That’s why management keeps pushing incentive pay: a lot of Managers are extrinsically motivated and they can’t get inside the head of someone who isn’t.

In case you happen to be a Manager, I feel it’s my duty to help you understand. It’s like this: if you’re addicted to flow, then being a Maker is an end in itself. If you still don’t get it, please read the story of The Fisherman and the Businessman. Ok, got it? Good.

So, if you’re managing people who are intrinsically motivated, here’s how you should setup their pay:

  • Make sure their salary or wage is competitive. If it’s not, they’ll feel cheated and resentful.
  • Profit sharing is ok if you don’t tell them how you arrived at the numbers.

Yeah, that’s crazy isn’t it? But it’s true. Do you want to know why? I’ve spent most of my career paid a straight wage without incentive pay. I did get bonuses, but they were just offered to me as, “good work this year, here’s X dollars, we hope you’re happy here, thanks very much.” Under that scheme when someone came to me with a problem, I relished the challenge. I dove into the problem, made sure I fully understood it, found the root cause and fixed it. The result was happy customers and higher quality work.

For a short time I was bullied into accepting performance-based pay. My metrics were “project gross margin” and “percent billable hours vs. target”. Then when someone came to me with a problem unrelated to my current project, my first reaction was “this is a distraction — I’m not being measured on this criteria.” When I paused to help a co-worker on another team for half an hour on their project, my boss reminded me that I wasn’t helping our team’s metrics. My demeanor with customers seemed to change. Work that used to seem challenging became extremely stressful. I lost all intrinsic motivation. I was no longer working to help the customer – I was working to screw the customer out of more money. It was as if, by dangling the carrot in front of my nose, I could no longer see the garden.

It’s hard for me to admit that. When you’re intrinsically motivated, you’re proud of it. It makes you feel good to do the work for its own sake. For Makers, doing work for performance pay is a hollow substitute. It demoralizes you.

I guess my point is, if you manage Makers, how do you not know this? It’s your job!

Ladder Logic vs. C#

PC programming and PLC programming are radically different paradigms. I know I’ve talked about this before, but I wanted to explore something that perplexes me… why do so many PC programmers hate ladder logic when they are first introduced to it? Ladder logic programmers don’t seem to have the same reaction when they’re introduced to a language like VB or C.

I mean, PC programmers really look down their noses at ladder logic. Here’s one typical quote:

Relay Ladder Logic is a fairly primitive langauge. Its hard to be as productive. Most PLC programmers don’t use subroutines; its almost as if the PLC world is one that time and software engineering forgot. You can do well by applying simple software engineering methods as a consequence, e.g., define interfaces between blocks of code, even if abstractly.

I’m sorry, but I don’t buy that. Ladder logic and, say C#, are designed for solving problems in two very different domains. In industrial automation, we prefer logic that’s easy to troubleshoot without taking down the system.

In the world of C#, troubleshooting is usually done in an offline environment.

My opinion is that Ladder Logic looks a lot like “polling” and every PC programmer knows that polling is bad, because it’s an inefficient use of processor power. PC programmers prefer event-driven programming, which is how all modern GUI frameworks react to user-initiated input. They want to see something that says, “when input A turns on, turn on output B”. If you’re familiar with control systems, your first reaction to that statement is, “sure, but what if B depends on inputs C, D, and E as well”? You’re right – it doesn’t scale, and that’s the first mistake most people make when starting with event-driven programming: they put all their logic in the event handlers (yeah, I did that too).

Still, there are lots of situations where ladder logic is so much more concise than say, C#, at implementing the same functionality, I just don’t buy all the hate directed at ladder logic. I decided to describe it with an example. Take this relatively simple ladder logic rung:

What would it take to implement the same logic in C#? You could say all you really need to write is D = ((A && B) || D) && C; but that’s not exactly true. When you’re writing an object oriented program, you have to follow the SOLID principles. We need to separate our concerns. Any experienced C# programmer will say that we need to encapsulate this logic in a class (let’s call it “DController” – things that contain business logic in C# applications are frequently called Controller or Manager). We also have to make sure that DController only depends on abstract interfaces. In this case, the logic depends on access to three inputs and one output. I’ve gone ahead and defined those interfaces:

    public interface IDiscreteInput
    {
        bool GetValue();
        event EventHandler InputChanged;
    }

    public interface IDiscreteOutput
    {
        void SetValue(bool value);
    }

Simple enough. Our controller needs to be able to get the value of an input, and be notified when any input changes. It needs to be able to change the value of the output.

In order to follow the D in the SOLID principles, we have to inject the dependencies into the DController class, so it has to look something like this:

    internal class DController
    {
        public DController(IDiscreteInput inputA, 
            IDiscreteInput inputB, IDiscreteInput inputC, 
            IDiscreteOutput outputD)
        {
        }
    }

That’s a nice little stub of a class. Now, as an experienced C# developer, I follow test-driven development, or TDD. Before I can write any actual logic, I have to write a test that fails. I break open my unit test suite, and write my first test:

        [TestMethod]
        public void Writes_initial_state_of_false_to_outputD_when_initial_inputs_are_all_false()
        {
            var mockInput = MockRepository.GenerateStub<IDiscreteInput>();
            mockInput.Expect(i => i.GetValue()).Return(false);
            var mockOutput = MockRepository.GenerateStrictMock<IDiscreteOutput>();
            mockOutput.Expect(o => o.SetValue(false));

            var test = new DController(mockInput, mockInput, mockInput, mockOutput);

            mockOutput.VerifyAllExpectations();
        }

Ok, so what’s going on here? First, I’m using a mocking framework called Rhino Mocks to generate “stub” and “mock” objects that implement the two dependency interfaces I defined earlier. This first test just checks that the first thing my class does when it starts up is to write a value to output D (in this case, false, because all the inputs are false). When I run my test it fails, because my DController class doesn’t actually call the SetValue method on my output object. That’s easy enough to remedy:

    internal class DController
    {
        public DController(IDiscreteInput inputA, IDiscreteInput inputB, 
            IDiscreteInput inputC, IDiscreteOutput outputD)
        {
            if (outputD == null) throw new ArgumentOutOfRangeException("outputD");
            outputD.SetValue(false);
        }
    }

That’s the simplest logic I can write to make the test pass. I always set the value of the output to false when I start up. Since I’m calling a method on a dependency, I also have to include a guard clause in there to check for null, or else my tools like ReSharper might start complaining at me.

Now that my tests pass, I need to add some more tests. My second test validates when my output should turn on (only when all three inputs are on). In order to write this test, I had to write a helper class called MockDiscreteInputPatternGenerator. I won’t go into the details of that class, but I’ll just say it’s over 100 lines long, just so that I can write a reasonably fluent test:

        [TestMethod]
        public void Inputs_A_B_C_must_all_be_true_for_D_to_turn_on()
        {
            MockDiscreteInput inputA;
            MockDiscreteInput inputB;
            MockDiscreteInput inputC;
            MockDiscreteOutput outputD;

            var tester = new MockDiscreteInputPatternGenerator()
                .InitialCondition(out inputA, false)
                .InitialCondition(out inputB, false)
                .InitialCondition(out inputC, false)
                .CreateSimulatedOutput(out outputD)
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputA).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputB).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputA).TurnsOff()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputC).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputB).TurnsOff()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputA).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputB).TurnsOn()
                .AssertThat(outputD).ShouldBe(true); // finally turns on

            var test = new DController(inputA, inputB, inputC, outputD);

            tester.Execute();
        }

What this does is cycle through all the combinations of inputs that don’t cause the output to turn on, and then I finally turn them all on, and verify that it did turn on in that last case.

I’ll spare you the other two tests. One check that the output initializes to on when all the inputs are on initially, and the last test checks the conditions that turn the output off (only C turning off, with A and B having no effect). In order to get all of these tests to pass, here’s my final version of the DController class:

    internal class DController
    {
        private readonly IDiscreteInput inputA;
        private readonly IDiscreteInput inputB;
        private readonly IDiscreteInput inputC;
        private readonly IDiscreteOutput outputD;

        private bool D; // holds last state of output D

        public DController(IDiscreteInput inputA, IDiscreteInput inputB, 
            IDiscreteInput inputC, IDiscreteOutput outputD)
        {
            if (inputA == null) throw new ArgumentOutOfRangeException("inputA");
            if (inputB == null) throw new ArgumentOutOfRangeException("inputB");
            if (inputC == null) throw new ArgumentOutOfRangeException("inputC");
            if (outputD == null) throw new ArgumentOutOfRangeException("outputD");

            this.inputA = inputA;
            this.inputB = inputB;
            this.inputC = inputC;
            this.outputD = outputD;

            inputA.InputChanged += new EventHandler((s, e) => setOutputDValue());
            inputB.InputChanged += new EventHandler((s, e) => setOutputDValue());
            inputC.InputChanged += new EventHandler((s, e) => setOutputDValue());

            setOutputDValue();
        }

        private void setOutputDValue()
        {
            bool A = inputA.GetValue();
            bool B = inputB.GetValue();
            bool C = inputC.GetValue();

            bool newValue = ((A && B) || D) && C;
            outputD.SetValue(newValue);
            D = newValue;
        }
    }

So if you’re just counting the DController class itself, that’s approaching 40 lines of code, and the only really important line is this:

    bool newValue = ((A && B) || D) && C;

It’s true that as you wrote more logic, you’d refactor more and more repetitive code out of the Controller classes, but ultimately most of the overhead never really goes away. The best you’re going to do is develop some kind of domain specific language which might look like this:

    var dController = new OutputControllerFor(outputD)
        .WithInputs(inputA, inputB, inputC)
        .DefinedAs((A, B, C, D) => ((A && B) || D) && C);

…or maybe…

    var dController = new OutputControllerFor(outputD)
        .WithInputs(inputA, inputB, inputC)
        .TurnsOnWhen((A, B, C) => A && B && C)
        .StaysOnWhile((A, B, C) => C);

…and how is that any better than the original ladder logic? That’s not even getting into the fact that you wouldn’t be able to use breakpoints in C# when doing online troubleshooting. This code would be a real pain to troubleshoot if the sensor connected to inputA was becoming flaky. With ladder logic, you can just glance at it and see the current values of A, B, C, and D.

Testing: the C# code is complex enough that it needs tests to prove that it works right, but the ladder logic is so simple, so declarative, that it’s obvious to any Controls Engineer or Electrician exactly what it does: turn on when A, B, and C are all on, and then stay on until C turns off. It doesn’t need a test!

Time-wise: it took me about a minute to get the ladder editor open and write that ladder logic, but about an hour to put together this C# example in Visual Studio.

So, I think when someone gets a hate on for ladder logic, it just has to be fear. Ladder logic is a great tool to have in your toolbox. Certainly don’t use ladder logic to write an ERP system, but do use it for discrete control.

On Budgets

What function do budgets serve?

In a case where there are well-known basic requirements, and highly variable-solutions, a budget is essential to curb the tendency to put wants ahead of needs. Take, for example, an automobile purchase. All you need is four wheels. Maybe you need four doors if you have two kids, or a minivan if you have three kids, but you certainly don’t need a sunroof, or a hemi, or a self-parking car. These are luxuries. Maybe you can afford to splurge a bit, but the budget knows there are other needs that take priority, like food, mortgage payments, and utilities.

When we try to take this concept of a budget to the workplace it breaks down. In the workplace, budgets serve one of two purposes: (1) flagging corruption, and (2) investment payback.

Purpose 1: Flagging Corruption

When we know about how much money it should take to do a job, we give the responsible person a budget. “Marty, we’ve been attending conferences in Las Vegas for 30 years. We know it costs about $X, so you have a budget of $X and $Y for contingency.” As long as Marty stays within that budget, keeps his receipts, and can avoid having the escort service charges showing up on his company Amex, he’s good to go. If his expense report comes in at twice the expected amount, then he has to justify it.

When you know how much something should cost, this is an efficient model. It delegates spending to the individual and only requires minor oversight from the accounting department.

Note that the budget wasn’t set based on how much money was in the bank, it was based on historical data, and the company decided there was a payback to send Marty to Las Vegas. That’s a fundamental difference between home and workplace budgets: at home we frequently buy stuff with no perceived payback, but at a company every expenditure is an investment. That’s why the function of budgets at home and at work are fundamentally different.

Purpose 2: Investment Paybacks

When you don’t have good historical data on a planned expenditure, accounting still needs an estimate. Estimates feedback expected expenditures to whoever is managing the cash flow. Budgets are the present-cost of the expected payback, with interest.

When a company looks at starting a new project, whether it’s done internally or subcontracted, first they get an estimate. Then they compare the estimate to the expected payback, and if it pays back within a certain time-frame (usually a couple of years, but sometimes as short as a few months), they go ahead with the project. However, since projects are, by definition, something you haven’t actually done before, everyone acknowledges that the estimate won’t necessarily be accurate. They will generally approve a budget with an added contingency. They’ve done the calculations to make sure that if the project comes in under that budget, the payback will make the company (and the investors) money.

As the project progresses, the project manager needs to track the actual expenditures against the expected expenditures, and provide updated estimates to accounting. If the estimate goes higher than the approved budget, management has to re-evaluate the viability of the project. They might increase the budget if the payback is good enough, or they might scrap the project.

Tying Incentives to Budgets

For home budgets, it’s implied: if you stay within your budget, you’ll make sure to satisfy all your needs, and you’re less likely to find yourself in financial hardship. You have an incentive to stay on budget.

At work, lots of people have their bonuses tied to “performance-to-budget”. If we’re talking about the first purpose of workplace budgets (flagging corruption) where historical data provides a solid prediction of what something should cost, then it makes sense to evaluate someone on their performance to that budget. On the other hand, if we accept that project budgets are based on rather inaccurate estimates, then measuring someone to a project budget isn’t very efficient.

In fact, it leads to all kinds of behaviors we want to avoid. First, people might tend to over-estimate projects because that will raise the budget. This might prevent the company from pursuing projects that could have been profitable. Secondly, project managers tend to play a “numbers game” – using resources from one project that’s going well to pad another project that’s going over. This destroys the accuracy of your project reports; now you won’t know which initiatives really made money. Third, the cost will be evaluated at the end of the project, but the payback continues over a longer time-frame. The project manager will tend to make choices that favor lower cost at the end of the project at the expense of choices that offer higher long-term payback.

Everything I’ve read suggests that (a) project managers have very little influence over the actual success or failure of a project and (b) in any task that requires creativity and out-of-the-box problem solving, offering a performance-incentive reduces performance because you’re replacing an intrinsic motivation with an extrinsic one. The intrinsic one is more powerful.

So why do we tie performance incentives to project budgets? Is there really any research that suggests this works, or are we just extrapolating what we know about purchasing a car, or taking a trip to Las Vegas? As someone who is intrinsically motived, I’ve found budget performance-incentives to be extremely distracting. Surely I’m not the only one, am I?

The Payback on Automated Unit Tests

I’m a Test-Driven Development (TDD) convert now. All of the “business logic” (aka “domain logic”), and more than 95% of my framework logic is covered by automated unit tests because I write the test before I write the code, and I only write enough code to pass the failing test.

It’s really hard to find anyone talking about a measurable ROI for unit testing, but it does happen. This study says it took, on average, 16% longer to develop the initial application using TDD than it did using normal development methods. This one reported “management estimates” of 15% to 35% longer development times using test-driven development. Both studies reported very significant reduction in defects. The implication is that the payback comes somewhere in the maintenance phase.

From personal experience I would say that as I gain experience with TDD, I get much faster at it. At the beginning it was probably doubling my development time, but now I’m closer to the estimates in the studies above. I’ve also shifted a bit from “whitebox testing” (where you test every little inner function) to more “blackbox testing”/”integration testing” where you test at a much higher level. I find that writing your tests at a higher level means you write fewer tests, and they’re more resilient to refactoring (when you change the design of your software later to accommodate new features).

A Long Term Investment

It’s hard to justify TDD because the extra investment of effort seems a lot more real and substantial than the rather flimsy value of quality. We can measure hours easily, but quality, not so much. That means we have a bias in our measurements.

Additionally, if and when TDD pays us back, it’s in the future. It’s probably not in this fiscal year. Just like procrastination, avoiding TDD pays off now. As humans, we’re wired to value immediate value over long term value. Sometimes that works against us.

A Theory of TDD ROI

I’m going to propose a model of how the ROI works in TDD. This is scientific, in that you’ll be able to make falsifiable predictions based on this model.

Start out with your software and draw out the major modules that are relatively separate from each other. Let’s say you’re starting with a simple CRUD application that just shows you data from a database and lets you Create, Read, Update, and Delete that data. Your modules might look like this:

  • Contact Management
  • Inventory Management

If you go and implement this using TDD vs. not using TDD, I suspect you’ll see a typical 15% to 35% increase in effort using the TDD methodology. That’s because the architecture is relatively flat and there’s minimal interaction. Contact Management and Inventory Management don’t have much to do with each other. Now let’s implement two more modules:

  • Orders
  • Purchasing

These two new modules are also relatively independent, but they both depend on the Contact Management and Inventory Management modules. That just added 4 dependency relationships. The software is getting more complex, and more difficult to understand the effect of small changes. The latter modules can still be changed relatively safely because nothing much depends on them, but the first two can start to cause trouble.

Now let’s add a Permissions module. Obviously this is a “cross cutting” concern – everything depends on the permissions module. Since we had 4 existing modules, we’ve just added another 4 dependency relationships.

Ok, now we’ll add a Reporting module. It depends on the 4 original modules, and it also needs Permissions information, so we’ve added another 5 dependency relationships.

Are you keeping count? We’re at 13 relationships now with just 6 modules.

Now let’s say we have to add a function that will find all customers (Contact Module) who have a specific product on order (Orders) that came from some manufacturer (Purchasing and Contact Management) and a certain Lot # (Inventory) and print a report (Reporting Module). Obviously this will only be available to certain people (Permissions).

That means you have to touch all 6 modules to make this change. Perhaps while you’re messing around in the Inventory Management module you notice that the database structure isn’t going to support this new feature. Maybe you have a many-to-one relationship where you realize we really should have used a many-to-many relationship. You change the database schema, and you change the Inventory Module, but instead of just re-testing that module, you now have to fully re-test all the modules that depend on it: Orders, Purchasing, and Reports. It’s likely we made assumptions about that relationship in those modules. What if we need to change those? Does the effect cascade to all the modules in the software? Likely.

It doesn’t take long to get to the point where you need to do a 100% regression test of your entire application. How many new features potentially touch all modules? How long does it take to do a full regression test? That’s your payback.

You can measure the regression test time, and if you use a tool like NDepend you can measure and graph the dependencies of an existing application. Using your source control history, you can go back and determine how many different modules were touched by each new feature and bug fix since the beginning of time. You should be able to calculate:

  • How much time it takes to regression test each module
  • Probability of each module changing during an “average” change
  • The set of modules to regression test for any given module changing

Given that, you can figure out the average time to regression test the average change.

Obviously, the average regression test time must be longer than 15% to 35% of the time it took to write the feature (assuming you’ll keep following TDD practices during the maintenance phase). The amount of time it takes to test in excess of that is payback against the initial 15% to 35% extra you spend developing the application in the first place.

What kind of numbers are we talking about?

Let’s run some numbers. A lot of places say 30 to 50% of software development time is spent testing. Let’s assume 50% is for the apps with “very interconnected dependencies”. Also, let’s say our team spends an extra 33% premium to use a TDD methodology.

Now take a project that would originally take 6 months to develop and test, but with TDD the development took about 33% longer, so +2 months. The average change takes 3 days to code and test, or 4 days with TDD. Let’s say the regression test on something that took 6 months to develop (personal experience – a 3 month project had a regression test plan that was about 1 day to run through) would have to be about 2 days.

Without TDD, a feature would take 3 days to write and test, and then 2 days to do a full regression test. Using TDD, it would take 4 days to write and test, but zero time to regression test, so you gain a day.

Therefore, since you had to invest an extra 2 months (40 days assuming one developer) in the first place to do TDD, you’d see a break-even once you were in the maintenance phase and had implemented about 40 changes, each taking 4 days, which means 160 days. That’s about 8 months. That ignores the fact that regression test time keeps increasing as you add more features.

Obviously your numbers will vary. The biggest factor is the ratio of regression test time vs. the time it takes you to implement a new feature. More dependencies means more regression testing.

Conclusion

If you have a very flat architecture with few dependencies, then the TDD payback is longer (if there even is a payback). On the other hand, if you have highly interdependent software (modules built on top of other modules), TDD pays you back quickly.

A Very Fast Tutorial on Open Source Licenses

I’ve written a bit of open source software lately, and I did a lot of learning about open source licenses. Unfortunately, after you learn a lot about a topic, you tend to subconsciously assume everyone knows what you do. In the interest of catching you all up, here’s a cheat sheet with some practical tips:

Note: I use the terms “proprietary” and “commercial” with specific meanings. “Commercial” means an application that isn’t open-source and you sell copies of it for money. “Proprietary” includes “commercial” but also includes software that is only used internally, never sold.

BSD or MIT/X11 Licenses

Sometimes called the “Academic” Licenses

Technically it’s the “revised” BSD license, but it’s been around so long that it’s just the BSD license now. It’s pretty much equivalent to the MIT/X11 license. These are considered to be the least restrictive, “do-anything-you-want” licenses as long as you keep the original notice on the code, display the copyright notice in your program, and don’t infer the author is endorsing your product. Also, it has a disclaimer saying they’re not responsible for anything you do with it. Standard stuff.

You can use this code in proprietary software, and you can use it in a GPL’d work (the common term for this is “GPL-compatible”.)

Apache 2.0

The Apache 2.0 license is similar in use to the BSD license, but it adds a patent grant. That is, it specifically states that if the authors hold any patents that cover the code, you can still use this code and be safe from that. Some might argue that BSD implies the patent grant, but it’s not specific, so Apache 2.0 makes it specific.

You can use it in proprietary software, and it is GPL-compatible, but only with version 3 of the GPL.

Mozilla Public License 1.1 and the Common Development and Distribution License

a.k.a. MPL 1.1 and CDDL

CDDL was based on the MPL 1.1.

These are your “weak copyleft” licenses, copyleft meaning some part of the derived work also has to be released under the same license. What it means is that you can use this code in a proprietary application, but if you make any changes to the code you included, you have to release that code publicly (but not the code of your entire work). The CDDL defines the boundary at the source file, so if you change one of the original files, you have to release that back.

These are not compatible with the GPL.

MS-PL

Microsoft Public License

You’ll see this license a lot if you do much .NET coding. A lot of the stuff you find on MS’s open source site, CodePlex, is MS-PL licensed. The short of it is that you can use it in proprietary applications, but it’s not GPL-compatible (by design – MS hates Linux). It’s also not copyleft at all.

GPL

Your strong copyleft

The GPL is the one everyone loves to hate, but it’s also popular because the Linux kernel is released under the GPLv2 license, and many/most of the tools of the Linux community are GPL-based.

Unlike the previous licenses, if you take any code from a GPL’d program and include it in your own project, you’re making a “derived work” and you must agree to release your entire derived work under the same license (or a later version if the author specifically says you can). That means you can’t use GPL’d code in your commercial application (but you can use it in internal applications).

The reason some programmers are so annoyed by it is that they’re at work Googling for some code to solve their problem, realize someone has written an open source library to do exactly what they want and they get all excited. Then they check the license and their heart drops because it’s GPL and they realize they can’t use it. A small minority go as far as to send hate mail to the author. (As someone who has released some GPL’d code, I’ve received my share of this hate mail, and I find it very silly. I’m offering something for free, under certain conditions, and you are free to take it or leave it.) What most programmers don’t seem to understand is that if you email the author, they’d probably be willing to sell you a commercial license for the code so you could use it in your program.

LGPL

The weak copyleft version

The “L” originally stood for library, but now it stands for “lesser”. It kind of works like the CDDL, but it defines the boundary at the “library” rather than the source file. It basically says you can use this library, even in a commercial application, but if you make any changes to it, you must release your new version of the library under the same (or newer…) license.

It also adds a restriction that some people overlook – you must also provide the users of your application the ability to replace the LGPL’d library with a newer or modified version of that library. Usually this means providing the binaries and compilation instructions. Consider the difficulty of meeting this obligation if your derived work is firmware on an embedded device. Version 3 of the GPL and LGPL make it very clear that you must allow your users all the tools needed to replace the software on the device. This was a reaction to the TiVo, which used GPL’d code, and released the code publicly, but didn’t allow anyone to further modify the code and update their TiVo’s with it.

The other thing you have to worry about is copying code. If you copy any code from the library into your main project, then your project becomes a derived work, and you’re essentially forced to release your whole application under the LGPL. Programmers don’t worry about this too much, but legal departments do.

AGPL

Affero?

It turns out you can take GPL’d code, run it on a server as a web application, make all the changes you want, and never release your code because you’re not “distributing” the derived work, and distribution is what triggers the GPL. (Google has it’s own version of Linux that it never had to release because it only uses it internally).

This ticked off some people who were writing GPL’d blogging and other website type software, so someone came up with the AGPL. This changes the triggering clause, so if you are using the AGPL’d code in a website, you have to make any changes public.

Conclusion

Those are the major licenses you’ll run into. If you’re writing commercial software, you want to look for BSD, MIT/X11, Apache 2.0, MS-PL, MPL 1.1 or CDDL code. You can also use LGPL’d code, but watch out for the extra restrictions.

If the proprietary software you’re writing is only for internal use, or you’re writing it “for hire” for another company that will only use it internally, then you’re safe to use GPL’d or LGPL’d code because you won’t trigger the distribution clause. Just be sure that you make this clear to your management/customer before you go down this path. If they decide they want to sell the software later, they’ll have a mess to clean up.

If you’re writing open source code then you need to pick a license. A BSD license is the easiest, and it’s great for little utility libraries because anyone can use it. If you’re writing an application and you want to protect against some company taking the application, adding a bunch of new features that make it incompatible, and then releasing and charging for it without ever giving you anything, then you should choose the GPL (or AGPL if it’s a web application).


On the Pathetic State of Automation Security

To start with, even PC security is pretty bad. Most programmers don’t seem to know the basic concepts for securely handling passwords (as the recent Sony data breach shows us). At least there are some standards, like the Payment Card Industry Data Security Standard.

Unfortunately, if PC security is a leaky bucket, then automation system security about as watertight as a pasta strainer. Here are some pretty standard problems you’re likely to find if you audit any small to medium sized manufacturer (and most likely any municipal facility, like, perhaps, a water treatment plant):

  • Windows PCs without up-to-date virus protection
  • USB and CD-rom (removable media) ports enabled
  • Windows PCs not set to auto-update
  • Remote access services like RDP or Webex always running
  • Automation PCs connected to the office network
  • Unsecured wireless access points attached to the network
  • Networking equipment like firewalls with the default password still set
  • PLCs on the office network, or even accessible from the outside!

All of these security issues have one thing in common: they’re done for convenience. It’s the same reason people don’t check the air in their tires or “forget” to change their engine oil. People are just really bad at doing things that are in their long term best interest.

Unfortunately, this security issue is becoming an issue of national security. Some have said there’s a “cyber-cold-war” brewing. After the news about Stuxnet, it’s pretty clear the war has turned “hot”.

I’m usually not a fan of regulations and over-reaching standards, but the fact is the Japanese didn’t build earthquake resistant buildings by individual choice. They did it because the building code required it. Likewise, I’ve seen a lot of resistance to the OSHA Machine Guarding standards because it imposes a lot of extra work on Control System Designers, and the companies buying automation, but I’m certain that we’re better off now that the standards are being implemented.

It’s time for an automation network security standard. Thankfully there’s one under development. ISA99 is the Industrial Automation and Control System Security Committee of ISA. A couple of sections of the new standard have already been published, but it’s not done yet. Also, you have to pay ISA a ransom fee to read it. I don’t think that’s the best way to get a standard out there and get people using it. I also think it’s moving very slowly. We all need to start improving security immediately, not after the committee gets around to meeting a few dozen more times.

I wonder if you could piece together a creative-commons licensed standard based on the general security knowledge already available on the internet…