There’s a lot of stuff that falls under the term “functional programming,” but I’m just going to focus on the “functional” part right now, meaning when you define the value of something as a function of something else.

In ladder logic, we define the values of internal state (internal coils or registers) and outputs. We define these as functions of the inputs and internal state. We call each function a “rung”, and one rung might look like this:

There’s something slightly odd going on in that rung though. You might say that we’ve defined C recursively, because C is a function of A, B, and itself. We all know, of course, that the PLC has no problem executing this code, and it executes as you would expect. That’s because the C on the right is not the same as the C on the left. The C on the right is the next state of C and the C on the left is the previous state of C.

Each time we scan, we redefine the value of C. That means C is an infinite time-series of true/false values. Huh?

Ok, imagine an array of true/false (boolean) values called “C”. The lower bound on the array index is zero, but the upper bound is infinite. C[0] is false (the value when we start the program). Then we start scan number 1, and we get to the rung above, and the PLC is really solving for is this:

If that were actually true (if it had an infinite array to store each coil’s value), then the ladder logic would be a truly functional programming language. But it’s not. Consider this:

In all modern PLCs, the first rung overwrites the value of C, so the second rung effectively uses the newly computed value for C when evaluating D. That means D[1] is defined as being equal to C[1] (the current state value of C). Why is this weird? Consider this:

By reversing the order of the rungs, I’ve changed the definition of D. After the re-ordering, D is now defined as C[0] (the previous state value of C) rather than C[1]. This isn’t a trivial difference. In an older PLC your scan time can be in the hundreds of milliseconds, so the D output can react noticeably slower in this case.

In a truly functional language, the re-ordering either wouldn’t be allowed (you can’t define D, which depends on C, before you define C) or the compiler would be able to determine the dependencies and re-order the evaluation so that C is evaluated before D. It would likely complain if it found a circular dependency between C and D, even though a PLC wouldn’t care about circular dependencies.

There are a few of reasons why PLCs are implemented like this. First, it saves memory. We would have to double our memory requirements if we always wanted to keep the last state and the next state around at the same time. Secondly, it’s easier to understand and troubleshoot. Not only does the PLC avoid keeping around two copies of each coil, but the programmer only has to worry about one value of each coil at any given point in the program. Third, the PLC runtime implementation is much simpler. It can be (and is) compiled to a kind of assembly language that can run efficiently on single threaded CPUs, which were the only CPUs available until recently.

Of course this comes with a trade-off. Imagine, for a moment, if rung-ordering didn’t matter. If you could solve the rungs in any order, that means you could also solve the rungs in parallel. That means if you upgraded to a dual-core CPU, you could instantly cut your scan time in half. Alas, the nature of ladder logic makes it very difficult to execute rungs in parallel.

On the other hand, we can still enforce a functional programming paradigm in our ladder logic programs if we follow these rules:

• Never define a coil more than once in your program.
• Don’t use a contact until after the rung where the associated coil has been defined.

That means there should only be one destructive write to any single memory location in your program. (It’s acceptable to use Set/Reset or a group of Move instructions that write to the same memory location as long as they’re on the same or adjacent rungs).

It also means that if coil C is defined on rung 5, then rungs 1 through 4 shouldn’t contain any contacts of coil C. This is the harder rule to follow. If you find you want to reference a coil before it’s defined, ask yourself if your logic couldn’t be re-organized to make it flow better.

Remember, someone trying to solve a problem in a PLC program starts at an output and uses cross references to move back through the program trying to understand it. Cross referencing from a contact to a coil that moves you forward in the program doesn’t require any logical leaps, but cross referencing to a coil later in the program means you need to logically think one scan backwards in time.

Benefits

While ladder logic isn’t a truly functional language, you can write ladder logic programs in the functional programming paradigm. If you do, you’ll find that your outputs react faster, and your programs are easier to understand and troubleshoot.

ladder-logic · plc

This interesting video about Stuxnet popped up on my Boxee Box today, and I thought I’d share it:

security · stuxnet

I write a lot about the PLC side of industrial automation, but it’s also fundamental to have a good foundation in the electrical side of things.

First of all, most modern (North American) industrial control system wiring diagrams have a relatively common numbering scheme, and once you understand the scheme, it makes it fairly easy to navigate the wiring diagram (commonly called a “print set”).

Let’s start with the page and line numbering. Most multi-page wiring diagrams use a two digit page number (page 1 is “01″). In the rare, but possible, event that you end up with over 99 pages, some diagrams will just add 3 digit page numbers (starting at “100″), but if there was any forethought, many designers will divide their wiring diagrams into sections, giving each section a letter (let’s say “A” for the header material, “B” for power distribution, “C” for safety circuits, etc.). Within each section, you can re-start the page numbering at “01″. This has the added bonus of letting you insert more pages into one section without messing up the page numbering.

Within a page, you’ll typically see line numbers down the left side (and frequently continuing down the middle if you don’t need the whole width of the page for your circuits). These numbers will start with the two digit page number, followed by a two digit line number. Typically these start at zero, and increment by twos:

• 1000
• 1002
• 1004
• 1006
• … and so on

Now, devices (like pushbuttons, power supplies, etc.) usually have a device ID based on the four digit line number where they are shown in the wiring diagram, possibly with a prefix or suffix noting the device type. So, if you have a pushbutton on line 2040, the device ID might be PB2040 or 2040PB. The device ID should also be attached to the device itself, normally with an indelible etched label (lamacoid). Therefore, if you find a device in the field, you should be able to find its location in the wiring diagram. (Finding the wiring diagram, of course, is often the more difficult task.)

Wires are numbered similarly. The wire number is typically based on the four digit line number where the wire starts, plus one extra digit or letter in case you have more than one wire number starting on the same line. So, the wire numbers for 2 wires starting on line 1004 might be 10041 and 10042.

It’s typical for wires to connect to devices that are on other pages (it’s extremely common, in fact). In that case, you’ll see off-page connectors. The shape of these vary based on whose standard was used for the wiring diagram, but they’re typically rectangles or hexagons. In either case, inside the shape will be the four digit line reference number where the wire continues. The other end of the off-page connector (on the other page) also has an off-page connector in the opposite direction. Note that you frequently see connectors from one place on a page to another place on the same page, if it happens to improve readability.

That’s all you really need to know to find devices and follow wires in a wiring diagram. Now, to understand the components in an industrial control system, that’s going to take longer than a blog post. For a great introduction, I recommend the book Industrial Motor Control by Stephen Herman. Google books has a great preview if you want to check it out.

wiring

A couple weeks ago I wrote a post about budgets, and Ken from over at Robot Shift replied:

My experience when compensation is set up to be win-win it’s a good thing. If someone’s working hard, and doing the right things often their impact far exceeds that of just their hourly contribution. It impacts teams, groups and profitability. If the organization profits from this impact, the employee should as well.

On the surface I think that’s true – we all appreciate profit sharing, but I think Ken’s talking about performance-metric-based-pay. I don’t agree with tying creative work to performance pay. The inner workings of the human mind continue to astound us. An article in New Scientist says:

…it may come as a shock to many to learn that a large and growing body of evidence suggests that in many circumstances, paying for results can actually make people perform badly, and that the more you pay, the worse they perform.

“In many circumstances” – I’m going to talk about one such circumstance shortly. Of course for more surprising information there’s the classic Dan Pink on the surprising science of motivation, and Clay Shirky’s talk on Human Motivation.

There’s a big difference between intrinsic motivation and extrinsic motivation. If you have to choose, intrinsic motivation is more powerful. It costs less, and it’s a stronger motive force. I believe that’s due to the phenomenon of “flow“. If you work in any type of creative field, you know the power of flow – that feeling you get when you’re 100% immersed in your work, the outside world seems to drift away, you lose track of time, but you’re incredibly productive. Without distractions you can focus your full attention on solving the problem at hand. Once you’ve experienced this a few times, it’s like a drug.

For someone like me, assuming I make enough money to pay the bills, I would trade any additional money to spend more time in “flow”. Yes, I’m Scott Whitlock and I’m addicted to flow. (I’m not the only one.) If you have one of those lucky days where you spend a full 8 hours without distractions, wrestling with big problems, you’ll come home mentally exhausted but exhilarated. At night you truly enjoy your family time, not because your job sucks and family time is an escape, but because you’re content. The next morning you’re excited on your drive to work. Your life has balance.

Paul Graham wrote an excellent essay about the Maker’s Schedule, Manager’s Schedule. The Maker blocks out their time in 4 hour chunks because they need to get into “flow” to do their best work. Managers, on the other hand, schedule their days in 30 or 60 minute chunks.

Do you see the problem? If a Manager never experiences flow then they can’t understand what motivates their Makers. That’s why management keeps pushing incentive pay: a lot of Managers are extrinsically motivated and they can’t get inside the head of someone who isn’t.

In case you happen to be a Manager, I feel it’s my duty to help you understand. It’s like this: if you’re addicted to flow, then being a Maker is an end in itself. If you still don’t get it, please read the story of The Fisherman and the Businessman. Ok, got it? Good.

So, if you’re managing people who are intrinsically motivated, here’s how you should setup their pay:

• Make sure their salary or wage is competitive. If it’s not, they’ll feel cheated and resentful.
• Profit sharing is ok if you don’t tell them how you arrived at the numbers.

Yeah, that’s crazy isn’t it? But it’s true. Do you want to know why? I’ve spent most of my career paid a straight wage without incentive pay. I did get bonuses, but they were just offered to me as, “good work this year, here’s X dollars, we hope you’re happy here, thanks very much.” Under that scheme when someone came to me with a problem, I relished the challenge. I dove into the problem, made sure I fully understood it, found the root cause and fixed it. The result was happy customers and higher quality work.

For a short time I was bullied into accepting performance-based pay. My metrics were “project gross margin” and “percent billable hours vs. target”. Then when someone came to me with a problem unrelated to my current project, my first reaction was “this is a distraction — I’m not being measured on this criteria.” When I paused to help a co-worker on another team for half an hour on their project, my boss reminded me that I wasn’t helping our team’s metrics. My demeanor with customers seemed to change. Work that used to seem challenging became extremely stressful. I lost all intrinsic motivation. I was no longer working to help the customer – I was working to screw the customer out of more money. It was as if, by dangling the carrot in front of my nose, I could no longer see the garden.

It’s hard for me to admit that. When you’re intrinsically motivated, you’re proud of it. It makes you feel good to do the work for its own sake. For Makers, doing work for performance pay is a hollow substitute. It demoralizes you.

I guess my point is, if you manage Makers, how do you not know this? It’s your job!

motivation

PC programming and PLC programming are radically different paradigms. I know I’ve talked about this before, but I wanted to explore something that perplexes me… why do so many PC programmers hate ladder logic when they are first introduced to it? Ladder logic programmers don’t seem to have the same reaction when they’re introduced to a language like VB or C.

I mean, PC programmers really look down their noses at ladder logic. Here’s one typical quote:

Relay Ladder Logic is a fairly primitive langauge. Its hard to be as productive. Most PLC programmers don’t use subroutines; its almost as if the PLC world is one that time and software engineering forgot. You can do well by applying simple software engineering methods as a consequence, e.g., define interfaces between blocks of code, even if abstractly.

I’m sorry, but I don’t buy that. Ladder logic and, say C#, are designed for solving problems in two very different domains. In industrial automation, we prefer logic that’s easy to troubleshoot without taking down the system.

In the world of C#, troubleshooting is usually done in an offline environment.

My opinion is that Ladder Logic looks a lot like “polling” and every PC programmer knows that polling is bad, because it’s an inefficient use of processor power. PC programmers prefer event-driven programming, which is how all modern GUI frameworks react to user-initiated input. They want to see something that says, “when input A turns on, turn on output B”. If you’re familiar with control systems, your first reaction to that statement is, “sure, but what if B depends on inputs C, D, and E as well”? You’re right – it doesn’t scale, and that’s the first mistake most people make when starting with event-driven programming: they put all their logic in the event handlers (yeah, I did that too).

Still, there are lots of situations where ladder logic is so much more concise than say, C#, at implementing the same functionality, I just don’t buy all the hate directed at ladder logic. I decided to describe it with an example. Take this relatively simple ladder logic rung:

What would it take to implement the same logic in C#? You could say all you really need to write is D = ((A && B) || D) && C; but that’s not exactly true. When you’re writing an object oriented program, you have to follow the SOLID principles. We need to separate our concerns. Any experienced C# programmer will say that we need to encapsulate this logic in a class (let’s call it “DController” – things that contain business logic in C# applications are frequently called Controller or Manager). We also have to make sure that DController only depends on abstract interfaces. In this case, the logic depends on access to three inputs and one output. I’ve gone ahead and defined those interfaces:

[csharp]
public interface IDiscreteInput
{
bool GetValue();
event EventHandler InputChanged;
}

public interface IDiscreteOutput
{
void SetValue(bool value);
}
[/csharp]

Simple enough. Our controller needs to be able to get the value of an input, and be notified when any input changes. It needs to be able to change the value of the output.

In order to follow the D in the SOLID principles, we have to inject the dependencies into the DController class, so it has to look something like this:

[csharp]
internal class DController
{
public DController(IDiscreteInput inputA,
IDiscreteInput inputB, IDiscreteInput inputC,
IDiscreteOutput outputD)
{
}
}
[/csharp]

That’s a nice little stub of a class. Now, as an experienced C# developer, I follow test-driven development, or TDD. Before I can write any actual logic, I have to write a test that fails. I break open my unit test suite, and write my first test:

[csharp]
[TestMethod]
public void Writes_initial_state_of_false_to_outputD_when_initial_inputs_are_all_false()
{
var mockInput = MockRepository.GenerateStub<IDiscreteInput>();
mockInput.Expect(i => i.GetValue()).Return(false);
var mockOutput = MockRepository.GenerateStrictMock<IDiscreteOutput>();
mockOutput.Expect(o => o.SetValue(false));

var test = new DController(mockInput, mockInput, mockInput, mockOutput);

mockOutput.VerifyAllExpectations();
}
[/csharp]

Ok, so what’s going on here? First, I’m using a mocking framework called Rhino Mocks to generate “stub” and “mock” objects that implement the two dependency interfaces I defined earlier. This first test just checks that the first thing my class does when it starts up is to write a value to output D (in this case, false, because all the inputs are false). When I run my test it fails, because my DController class doesn’t actually call the SetValue method on my output object. That’s easy enough to remedy:

[csharp]
internal class DController
{
public DController(IDiscreteInput inputA, IDiscreteInput inputB,
IDiscreteInput inputC, IDiscreteOutput outputD)
{
if (outputD == null) throw new ArgumentOutOfRangeException("outputD");
outputD.SetValue(false);
}
}
[/csharp]

That’s the simplest logic I can write to make the test pass. I always set the value of the output to false when I start up. Since I’m calling a method on a dependency, I also have to include a guard clause in there to check for null, or else my tools like ReSharper might start complaining at me.

Now that my tests pass, I need to add some more tests. My second test validates when my output should turn on (only when all three inputs are on). In order to write this test, I had to write a helper class called MockDiscreteInputPatternGenerator. I won’t go into the details of that class, but I’ll just say it’s over 100 lines long, just so that I can write a reasonably fluent test:

[csharp]
[TestMethod]
public void Inputs_A_B_C_must_all_be_true_for_D_to_turn_on()
{
MockDiscreteInput inputA;
MockDiscreteInput inputB;
MockDiscreteInput inputC;
MockDiscreteOutput outputD;

var tester = new MockDiscreteInputPatternGenerator()
.InitialCondition(out inputA, false)
.InitialCondition(out inputB, false)
.InitialCondition(out inputC, false)
.CreateSimulatedOutput(out outputD)
.AssertThat(outputD).ShouldBe(false)

.Then(inputA).TurnsOn()
.AssertThat(outputD).ShouldBe(false)

.Then(inputB).TurnsOn()
.AssertThat(outputD).ShouldBe(false)

.Then(inputA).TurnsOff()
.AssertThat(outputD).ShouldBe(false)

.Then(inputC).TurnsOn()
.AssertThat(outputD).ShouldBe(false)

.Then(inputB).TurnsOff()
.AssertThat(outputD).ShouldBe(false)

.Then(inputA).TurnsOn()
.AssertThat(outputD).ShouldBe(false)

.Then(inputB).TurnsOn()
.AssertThat(outputD).ShouldBe(true); // finally turns on

var test = new DController(inputA, inputB, inputC, outputD);

tester.Execute();
}
[/csharp]

What this does is cycle through all the combinations of inputs that don’t cause the output to turn on, and then I finally turn them all on, and verify that it did turn on in that last case.

I’ll spare you the other two tests. One check that the output initializes to on when all the inputs are on initially, and the last test checks the conditions that turn the output off (only C turning off, with A and B having no effect). In order to get all of these tests to pass, here’s my final version of the DController class:

[csharp]
internal class DController
{
private readonly IDiscreteInput inputA;
private readonly IDiscreteInput inputB;
private readonly IDiscreteInput inputC;
private readonly IDiscreteOutput outputD;

private bool D; // holds last state of output D

public DController(IDiscreteInput inputA, IDiscreteInput inputB,
IDiscreteInput inputC, IDiscreteOutput outputD)
{
if (inputA == null) throw new ArgumentOutOfRangeException("inputA");
if (inputB == null) throw new ArgumentOutOfRangeException("inputB");
if (inputC == null) throw new ArgumentOutOfRangeException("inputC");
if (outputD == null) throw new ArgumentOutOfRangeException("outputD");

this.inputA = inputA;
this.inputB = inputB;
this.inputC = inputC;
this.outputD = outputD;

inputA.InputChanged += new EventHandler((s, e) => setOutputDValue());
inputB.InputChanged += new EventHandler((s, e) => setOutputDValue());
inputC.InputChanged += new EventHandler((s, e) => setOutputDValue());

setOutputDValue();
}

private void setOutputDValue()
{
bool A = inputA.GetValue();
bool B = inputB.GetValue();
bool C = inputC.GetValue();

bool newValue = ((A && B) || D) && C;
outputD.SetValue(newValue);
D = newValue;
}
}
[/csharp]

So if you’re just counting the DController class itself, that’s approaching 40 lines of code, and the only really important line is this:

[csharp]
bool newValue = ((A && B) || D) && C;
[/csharp]

It’s true that as you wrote more logic, you’d refactor more and more repetitive code out of the Controller classes, but ultimately most of the overhead never really goes away. The best you’re going to do is develop some kind of domain specific language which might look like this:

[csharp]
var dController = new OutputControllerFor(outputD)
.WithInputs(inputA, inputB, inputC)
.DefinedAs((A, B, C, D) => ((A && B) || D) && C);
[/csharp]

…or maybe…

[csharp]
var dController = new OutputControllerFor(outputD)
.WithInputs(inputA, inputB, inputC)
.TurnsOnWhen((A, B, C) => A && B && C)
.StaysOnWhile((A, B, C) => C);
[/csharp]

…and how is that any better than the original ladder logic? That’s not even getting into the fact that you wouldn’t be able to use breakpoints in C# when doing online troubleshooting. This code would be a real pain to troubleshoot if the sensor connected to inputA was becoming flaky. With ladder logic, you can just glance at it and see the current values of A, B, C, and D.

Testing: the C# code is complex enough that it needs tests to prove that it works right, but the ladder logic is so simple, so declarative, that it’s obvious to any Controls Engineer or Electrician exactly what it does: turn on when A, B, and C are all on, and then stay on until C turns off. It doesn’t need a test!

Time-wise: it took me about a minute to get the ladder editor open and write that ladder logic, but about an hour to put together this C# example in Visual Studio.

So, I think when someone gets a hate on for ladder logic, it just has to be fear. Ladder logic is a great tool to have in your toolbox. Certainly don’t use ladder logic to write an ERP system, but do use it for discrete control.

ladder-logic · productivity

What function do budgets serve?

In a case where there are well-known basic requirements, and highly variable-solutions, a budget is essential to curb the tendency to put wants ahead of needs. Take, for example, an automobile purchase. All you need is four wheels. Maybe you need four doors if you have two kids, or a minivan if you have three kids, but you certainly don’t need a sunroof, or a hemi, or a self-parking car. These are luxuries. Maybe you can afford to splurge a bit, but the budget knows there are other needs that take priority, like food, mortgage payments, and utilities.

When we try to take this concept of a budget to the workplace it breaks down. In the workplace, budgets serve one of two purposes: (1) flagging corruption, and (2) investment payback.

Purpose 1: Flagging Corruption

When we know about how much money it should take to do a job, we give the responsible person a budget. “Marty, we’ve been attending conferences in Las Vegas for 30 years. We know it costs about $X, so you have a budget of $X and $Y for contingency.” As long as Marty stays within that budget, keeps his receipts, and can avoid having the escort service charges showing up on his company Amex, he’s good to go. If his expense report comes in at twice the expected amount, then he has to justify it.

When you know how much something should cost, this is an efficient model. It delegates spending to the individual and only requires minor oversight from the accounting department.

Note that the budget wasn’t set based on how much money was in the bank, it was based on historical data, and the company decided there was a payback to send Marty to Las Vegas. That’s a fundamental difference between home and workplace budgets: at home we frequently buy stuff with no perceived payback, but at a company every expenditure is an investment. That’s why the function of budgets at home and at work are fundamentally different.

Purpose 2: Investment Paybacks

When you don’t have good historical data on a planned expenditure, accounting still needs an estimate. Estimates feedback expected expenditures to whoever is managing the cash flow. Budgets are the present-cost of the expected payback, with interest.

When a company looks at starting a new project, whether it’s done internally or subcontracted, first they get an estimate. Then they compare the estimate to the expected payback, and if it pays back within a certain time-frame (usually a couple of years, but sometimes as short as a few months), they go ahead with the project. However, since projects are, by definition, something you haven’t actually done before, everyone acknowledges that the estimate won’t necessarily be accurate. They will generally approve a budget with an added contingency. They’ve done the calculations to make sure that if the project comes in under that budget, the payback will make the company (and the investors) money.

As the project progresses, the project manager needs to track the actual expenditures against the expected expenditures, and provide updated estimates to accounting. If the estimate goes higher than the approved budget, management has to re-evaluate the viability of the project. They might increase the budget if the payback is good enough, or they might scrap the project.

Tying Incentives to Budgets

For home budgets, it’s implied: if you stay within your budget, you’ll make sure to satisfy all your needs, and you’re less likely to find yourself in financial hardship. You have an incentive to stay on budget.

At work, lots of people have their bonuses tied to “performance-to-budget”. If we’re talking about the first purpose of workplace budgets (flagging corruption) where historical data provides a solid prediction of what something should cost, then it makes sense to evaluate someone on their performance to that budget. On the other hand, if we accept that project budgets are based on rather inaccurate estimates, then measuring someone to a project budget isn’t very efficient.

In fact, it leads to all kinds of behaviors we want to avoid. First, people might tend to over-estimate projects because that will raise the budget. This might prevent the company from pursuing projects that could have been profitable. Secondly, project managers tend to play a “numbers game” – using resources from one project that’s going well to pad another project that’s going over. This destroys the accuracy of your project reports; now you won’t know which initiatives really made money. Third, the cost will be evaluated at the end of the project, but the payback continues over a longer time-frame. The project manager will tend to make choices that favor lower cost at the end of the project at the expense of choices that offer higher long-term payback.

Everything I’ve read suggests that (a) project managers have very little influence over the actual success or failure of a project and (b) in any task that requires creativity and out-of-the-box problem solving, offering a performance-incentive reduces performance because you’re replacing an intrinsic motivation with an extrinsic one. The intrinsic one is more powerful.

So why do we tie performance incentives to project budgets? Is there really any research that suggests this works, or are we just extrapolating what we know about purchasing a car, or taking a trip to Las Vegas? As someone who is intrinsically motived, I’ve found budget performance-incentives to be extremely distracting. Surely I’m not the only one, am I?

budget · estimation · motivation · project-management

To start with, even PC security is pretty bad. Most programmers don’t seem to know the basic concepts for securely handling passwords (as the recent Sony data breach shows us). At least there are some standards, like the Payment Card Industry Data Security Standard.

Unfortunately, if PC security is a leaky bucket, then automation system security about as watertight as a pasta strainer. Here are some pretty standard problems you’re likely to find if you audit any small to medium sized manufacturer (and most likely any municipal facility, like, perhaps, a water treatment plant):

• Windows PCs without up-to-date virus protection
• USB and CD-rom (removable media) ports enabled
• Windows PCs not set to auto-update
• Remote access services like RDP or Webex always running
• Automation PCs connected to the office network
• Unsecured wireless access points attached to the network
• Networking equipment like firewalls with the default password still set
• PLCs on the office network, or even accessible from the outside!

All of these security issues have one thing in common: they’re done for convenience. It’s the same reason people don’t check the air in their tires or “forget” to change their engine oil. People are just really bad at doing things that are in their long term best interest.

Unfortunately, this security issue is becoming an issue of national security. Some have said there’s a “cyber-cold-war” brewing. After the news about Stuxnet, it’s pretty clear the war has turned “hot”.

I’m usually not a fan of regulations and over-reaching standards, but the fact is the Japanese didn’t build earthquake resistant buildings by individual choice. They did it because the building code required it. Likewise, I’ve seen a lot of resistance to the OSHA Machine Guarding standards because it imposes a lot of extra work on Control System Designers, and the companies buying automation, but I’m certain that we’re better off now that the standards are being implemented.

It’s time for an automation network security standard. Thankfully there’s one under development. ISA99 is the Industrial Automation and Control System Security Committee of ISA. A couple of sections of the new standard have already been published, but it’s not done yet. Also, you have to pay ISA a ransom fee to read it. I don’t think that’s the best way to get a standard out there and get people using it. I also think it’s moving very slowly. We all need to start improving security immediately, not after the committee gets around to meeting a few dozen more times.

I wonder if you could piece together a creative-commons licensed standard based on the general security knowledge already available on the internet…

isa99 · security · standards · stuxnet

I’d like to share with you an excerpt from a TED talk by Ed Burtynsky, a Canadian photographer and filmmaker. TED licenses its talks under the Creative Commons “Attribution – NonCommercial – NonDerivative” license, which means it’s not legal to take an excerpt from one of their talks and edit it in any form. However, I must not be the first person to feel this particular clip is worth highlighting, because someone has already excerpted this one clip and uploaded it to YouTube. Here it is… a video of a Chinese woman, working in a factory, assembling a circuit breaker in just under 45 seconds:

That video has been up there for a few years now, but in case it gets taken down, and just to give proper attribution, you can go and watch the original talk on Manufactured Landscapes.

The video of the woman is very interesting to someone in automation. I’ve been in a lot of different factories, and I’ve seen people working hard everywhere, but I’ve never seen anything close to that. The skill is impressive. It reminds me of a soldier tearing down their rifle for cleaning and putting it back together. The motions are completely automatic. She’s using the part of the brain that an experienced driver uses when going to work every day.

When we think of Chinese manufacturing, I think we have a vision of poor quality and poor working conditions. I wouldn’t want a factory job in China, but I admire this person. She’s good at her job. She’s productive. While I’m sure we could build a machine that could automate this assembly process, automation can’t compete economically with this person. As time passes, Chinese wages will rise, and automation will become more flexible and less expensive. I have a vision of this woman working next to a machine, racing it in one last competition pitting human vs. machine in a battle of dexterity and speed, much like the fable of John Henry against the steam powered hammer.

That’s actually a happy ending. China’s unemployment rate will be down, living standards will be up, and new, more interesting opportunities will be available.

It makes you wonder though, if you worked that hard 10 hours a day, with the tools that are at your fingertips, what could you achieve?

productivity

If you’re going to interview for a control systems job in a plant, they’ll ask you a lot of questions, but you should also have some questions for them. To me, these are the minimum questions you need to ask to determine if a future employer is worth pursuing:

1. Do you have up-to-date electrical drawings in every electrical panel? – When the line is down, you don’t have time to go digging.
2. Do you have a wireless network throughout the plant? – It should go without saying, having good reliable wireless connectivity all over your facility really helps when you’re troubleshooting issues. Got a problem with a sensor? Just setup your laptop next to the sensor, go online, look at the logic, and flag the sensor. You don’t have time to walk all over.
3. Does every PC (including on-machine PCs) have virus protection that updates automatically? – We’re living in a post Stuxnet world. Enough said.
4. Have you separated the office network from the industrial network? – Protection and security are applied in layers. There’s no need for Jack and Jill in accounting to be pinging your PLCs.
5. What is your backup (and restore) policy? – Any production-critical machine must always have up-to date code stored in a known location (on a server or in a source control system), it must be backed up regularly, and you have to test your backups by doing regular restores.
6. Are employees compensated for working extra hours? – Nothing raises a red flag about a company’s competency more than expecting 60+ hour weeks but not paying you overtime. It means they’re reactive, not proactive. It means they don’t value experience (experienced employees have families and can’t spend as much time at the office). It probably means they scored poorly in the previous questions.

You don’t have to find a company that gets perfect on this test, but if they miss more than one or two, that’s a warning sign. If they do well, they’re a proactive company, and proactive companies are sane places to work.

Good luck!

employment · quality · risk

I’ve been enjoying a lot more TED recently now that I can stream it directly to our living room HDTV on our Boxee Box. Today it surprised me with a talk by Ralph Langner called “Cracking Stuxnet: A 21st Century Cyber Weapon”. I talked about Stuxnet before, but this video has even more juicy details. If you work with PLCs, beware; this is the new reality of industrial automation security:

plc · security · stuxnet