I think that once you’ve been in this industry for a few years, you need to reach out to others to share some of the wisdom you’ve learned. Most of the knowledge we carry around can do other people a lot more good than it will do us again in the future, so sharing needs to be a cultural norm. With that thought in mind, here are some quick automation-related thoughts I’d like to share:

1. Inexperienced engineers appear to work faster, but their solutions are less maintainable. [tweet this]
2. Choose open systems over proprietary, when possible [tweet this]
3. Automate your own job ruthlessly before to automate anything else. It pays back. [tweet this]
4. Beware of employers who spend 30 minutes reprimanding you about a 15 minute line on your timesheet. [tweet this]
5. If you can’t find a more powerful tool, make your own. [tweet this]
6. When estimating a project, if you’re counting in hours, you’re not being realistic. Use half-days. [tweet this]
7. Don’t take shortcuts writing a program if it’s at the expense of readability. It doesn’t pay off. [tweet this]
8. Automation doesn’t help if you don’t understand the process you’re automating. [tweet this]
9. Blame is reactive. “What can we do differently next time?” is proactive. [tweet this]
10. Innovate is a verb. This is not a coincidence – it requires constant action. [tweet this]
11. Make things of value, not emails. [tweet this]

Feel free to share your own nuggets of wisdom below.

deep-thoughts

We’re all probably familiar with the idea that it takes half the time to get to 90% done and the other half to finish the last 10%. This is a staple of project management.

I think there’s actually a narrower scope of really dangerous solutions that you only become familiar with after you experience it. There’s a whole set of problems where the obvious solution gets you 95 to 98% of the way to your performance spec really quickly, but is almost impossible to reach 100% by incremental improvements. The reason I say they’re dangerous is because the feeling of being “almost there” prevents you from going back to the drawing board and coming up with a completely parallel solution.

I can remember a machine vision job from years ago where the spec was “100% read rate”. I only got it to about 94%, and someone else gave it a try. He got it up over 96%, but 100% was out of reach given the technology we had.

Experiences like that make you conservative. Now I unconsciously filter possible solutions by their apparent “flakiness”. I’m much more likely to add an extra prox to a solution to verify a position than to rely on timers or other kinds of internal state, because the latter are more prone to failure during system starts and stops. I press for mechanical changes when I used to bend under the pressure to “fix it in software”.

Still, you have to be careful. Its easy to discount alternatives just because they bear some passing resemblance to a bad experience you had before. You have to keep re-checking your assumptions. Unfortunately, rapid prototyping usually fails to uncover the “almost there” situation I’m talking about. If you prototype something up fast, and it works in 97% of your lab tests, you’ll probably think you have a “proof of concept”, and go forward with it.

The best way to test new solutions is to put them into production on a low risk system. If you’re an integrator, this means having a really good relationship with your customer (chances are you need their equipment to run your tests). If you work for a manufacturer, you can usually find some out-of-the-way machine to test on before you go all-in.

quality · risk

I’ve just been reading Ken McLaughlin’s recent post Top Ten Signs an Integrator is the Real Deal #7: Best Practices and Standards and I have to say, my initial reaction is one of skepticism. I think Ken’s thinking is a little too narrow on this one. Let me explain…

This isn’t the first time I’ve considered the “problem of standards” on this blog. In an earlier post, Standards for the Sake of Standards, I explained how most corporate standards eventually end up being out-of-date and absurd, mostly because nobody making the standard ever things to write down Why the standard exists, which would allow future policy-makers to understand the reasons and change the standard when it no longer applied. Instead, it becomes gospel.

However, that isn’t to say you could run a large organization without best practices and standards. That’s the point isn’t it? In order to become large, you need built-in efficiency, and you do that at the expense of innovation. Big companies don’t innovate (in fact the only notable exception is Apple, and the rebuttal is always, “fine, so give one example other than Apple”). Almost all innovation happens in small companies, by a tightly knit group of superstars where the chains have been removed. Best Practices are, in fact, put in place to clamp down on innovation because innovation is risky, and investors hate risk. It’s better to make lots of average product for average people than exceptional products for a few people (hence McDonald’s). Paul Graham, as usual, has something insightful to add to this:

Within large organizations, the phrase used to describe this approach is “industry best practice.” Its purpose is to shield the pointy-haired boss from responsibility: if he chooses something that is “industry best practice,” and the company loses, he can’t be blamed. He didn’t choose, the industry did.

I believe this term was originally used to describe accounting methods and so on. What it means, roughly, is don’t do anything weird. And in accounting that’s probably a good idea. The terms “cutting-edge” and “accounting” do not sound good together. But when you import this criterion into decisions about technology, you start to get the wrong answers.

The reason small companies are innovative is that innovative people can’t stand corporate environments. Imagine if you were an inspired chef… could you stand working at McDonald’s? Could McDonald’s even stand to employ you? You’d be too much trouble! You’d have to work in that nice one-off restaurant called “Maison d’here” where the manager puts up with your off-beat attitude because ultimately you make good food, and you keep their small but devoted clientèle coming back. But you can’t be franchised. The manager of the restaurant can’t scale you up without making what you do into a procedure.

So back to Ken’s topic… if you are choosing a systems integrator, you need to decide if you’re buying an accounting system (i.e. something that’s generic to all companies, and not a competitive advantage), or something that is a competitive advantage to you. When you’re automating your core business processes, you must build competitive advantage into it, and it must be innovative. If that’s the case, stay away from larger integrators with miles and miles of red tape and bureaucracy. Go for the “boutique” integrator (somewhere in the 7 to 25 person sized company, under $10 million per year in revenue) that can show you good references. You’re looking for a small group of passionate people. Buzzwords are a warning sign; small companies don’t have time for corporate-speak.

I’m not saying you should use the two guys in their garage. These guys are ok for your basic maintenance tasks, small changes, and local support, but you do want someone who has been around for a few years and has at least a couple of backup engineers they can pull in if there’s a problem. Make sure they have a server, with backups, and all that.

On the other hand, if what you’re automating is very large and very standard, that’s when you want to go with Ken’s approach. If you need to integrate a welding line, paint line, or whatever, there’s nothing new or innovative in that, so you want to lower the risk. You know all the big integration companies can do this, so go and get three bids, and choose the one that’s hungriest for the work. Make sure they have standards and best practices. The reduction in risk is worth it if you don’t need the innovative solution.

You can do a hybrid approach. Identify the parts of your process that could be key competitive advantages if you could find a better way to do it. This is where innovation pays off. Go out and consult with some boutique integrators ahead of time and get them working on those “point solutions”. Then go to the bigger companies to farm out the rest of your automation needs. How’s that for a “best practice”?

innovation · standards

Start by watching this video about the Aeryon Scout robot (kudos Kareem):

I think what sets this aerial robot apart, as Kareem says, is the intuitive user interface. When I look at the state of automation today, I can see that good user interfaces are typically an after-thought. Custom solutions are sometimes so cobbled together that there isn’t enough bandwidth between one black box and the HMI, or the HMI is just a simple two line text display that ends up saying FAULT 53 (the manual with the list of faults, of course, is stuck inside the door of the panel, and it’s the only thing in the area that isn’t covered in grease because nobody bothers to look at it).

People frequently blame engineers for this mess, which I find a bit silly. Certainly user interfaces are a critical component of any system, but why do you hire an electrical designer to do the electrical design, hire a programmer to write the software, but expect one of these people to magically become a usability expert, which is a field unto itself?

I think there used to be an idea that there was no payback on usability. Certainly if you’re selling something like a VCR, you can only print features on the box (you can’t accurately represent the experience of using it) and people only buy one. However, as items become more social (think iPhone), we’re starting to see great user interfaces create viral marketing for products. I think I first saw this with the TiVo – once you saw what it could do, and how easy it was, you were hooked. Apple’s technology seems to be the same way, and I can see how the Aeryon Scout probably has the same “shock and awe” effect when you demo it.

Where does that leave us with industrial automation interfaces? Automation is always purchased based on a cost-benefit analysis because of the high capital cost. The operators typically don’t participate in the purchasing decision at all. I don’t think effort put into a better user interface is wasted; in fact I’m certain there’s a long term payback. But it’s not a selling feature and it takes more time to do right.

Still, when I programmed a machine recently, it was nice to overhear someone say, “it’s pretty intuitive, isn’t it?” So I guess I’ll keep trying, even if it’s not in my own best interest. Engineers are weird that way.

If you’re interested in making better user interfaces, first I recommend reading The Design of Everyday Things by Donald Norman. I also recommend this video called the least you can do about usability by Steve Krug, author of Don’t Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition.

hmi · quality · usability

As we all know, there are 10 kinds of people in the world.

For those of you who haven’t read Zen and the Art of Motorcycle Maintenance by Pirsig, he spends at least one chapter at the beginning talking about how we naturally tend to divide things into smaller pieces in an effort to understand them. The novice looks at a motorcycle and sees the visible things, like a seat, handlebars, and wheels, but the expert sees a fuel system, a cooling system, and the suspension. The same thing or system (motorcycle) can be subdivided different ways depending on what we want to do with it.

My tongue-in-cheek title of this post is an acknowledgement of the many ways we can categorize something like Automation Software, but for my purposes today, I’m making two categories: hammers and levels.

A carpenter carries both a hammer and a level, but the two have fundamentally different failure modes. If a hammer stops working, you’ll know it as soon as you try to use it. As long as it hammers in a nail, it doesn’t matter if the hammer is rusty, dirty, scratched or dented, it’s a working hammer. The level, on the other hand, is a measuring instrument. As novices, we assume that it comes from the factory pre-calibrated, and we happily hang our shelf or picture without testing it, but a professional carpenter knows that they have to check their levels for accuracy, or else the level is useless. You could use a level for years, but if one day it stopped being accurate, you probably wouldn’t know. This is a very different situation than the hammer.

Software in general, and automation software in particular, both have similar examples. You never need to “calibrate” the Axis 1 Advanced proximity switch on a machine because if it doesn’t work, the machine won’t make parts (and you’ll know about it instantly, usually via a 2 am phone call). On the other hand, testing data collection logic is surprisingly difficult because the only way to test it is to compare it with a known-good equivalent. Assuming you created this data collection logic to automate away a manual process, the only measuring stick we can check it against is the manual process we’re replacing. Once the system is bought off and we get rid of the paper system, how do you prove that subsequent changes don’t break the data collection system?

It’s tempting to brush off the problem by saying that anyone who makes a subsequent change has to do a full regression test of the system, including the data collection system, but anyone who has worked in a real factory environment knows that this is unlikely to work in practice. Full regression tests are expensive.

In the greater software world, they use automated unit tests. They take the logic being tested and they run it through a series of automated checks to make sure nothing changes. This works well in an environment like PC programming, but is very difficult in practice for PLC programming because (a) you usually need a physical PLC to execute the logic (unless you have some kind of emulator) and (b) the people maintaining the system are likely not familiar with concepts like unit tests, and are likely to undervalue their importance.

This screams for a system-level solution. Take accounting for instance. Double-entry accounting (the use of debits and credits to force every action to be made twice) is deliberately created to help catch manual entry errors. If your debits and credits don’t balance, you know you’ve made a mistake somewhere, and you go back and check your arithmetic.

In the automation world, the solution is to measure every input to the data collection system two ways, analyze and aggregate both separately, and compare the end results. Create a system warning or fault if the results don’t match. For instance, measure the amount of material going into the machine, and measure the amount of material exiting the machine, both as finished product, and scrap. If the input doesn’t match the sum of the outputs over the same time period, you know you have a problem. The system becomes self-checking (a hammer rather than a level).

If you follow this route, you need to take care to avoid some common traps:

• Don’t re-use logic between the two sides (in fact, try to make them work differently)
• Try to use different sensors or sensing methods (can we measure the input by speed and duration, and the output by parts and scrap weight?)
• Record both, so if there is a discrepancy, you can check them against manual measurements

It sounds like more work, but making the system self-checking actually reduces the amount of testing you have to do, so it’s not as bad as you think. Besides, writing code is a lot more fun than testing it. We automate everyone else’s job, why not the boring parts of ours?

plc · quality · troubleshooting

I recently wrote part 11 of the RSLogix 5000 tutorial I’ve been working on, and this part deals with the Sequential Function Chart editor.

I know there’s a lot of resistance to straying away from the established usage of ladder logic everywhere, but in this part of the tutorial I present a really simple way to use sequential function charts to express auto mode sequences in your RSLogix 5000 program that’s very readable, saves a lot of ladder programming, and integrates very well with your lower level machine control routines. Even if you’re an experienced RSLogix 5000 programmer, you’ll find this SFC introduction worth the read. Check it out.

rslogix-5000 · sfc

I’m intrigued by the idea of using this technology to align, sort, inspect, and separate parts moving down a conveyor, or even falling down a chute:

innovation

Since I’m in the tutorial mind-set right now, I thought I’d mention this little gem. Here’s how you can read the ControlLogix (or CompactLogix) PLC system time into a UDT so you can use the current time value in your ladder logic program.

First I created a UDT called “TIME”:

Then you just need to use the GSV (Get System Value) instruction with the WallClockTime class, and LocalSystemTime attribute to read the controller’s time into an instance of your UDT (here I created a new tag called LocalDateTime of type TIME). Note that I used the Year element of the LocalDateTime tag as the parameter, because that’s the first address of the tag. It starts writing there but fills in the entire UDT with the time values:

Now you can program your sprinklers to turn on and off in the middle of the night!

plc · rslogix-5000

I’m still working on my RSLogix 5000 Tutorial, and I decided to include a brief introduction to a Five Rung Logic Block. If you’re like me, the first time you saw or heard about a five rung, you thought it was was a ludicrous amount of logic just to move a cylinder from one position to another. However, I realized most of my non-five-rung motion control logic always ended up being as complicated, or more complicated, than if I’d used a Five Rung from the beginning, so now I’m a convert.

A five rung block is named after the five coils that make up each block (these names differ between programmers and organizations, but they follow this pattern):

• Safety
• Precondition (aka Trigger)
• Command
• Complete (aka “In Position”)
• Fault

Briefly, your safety rung contains all the conditions that must be present during the entire motion. You put conditions like “no critical faults” here. You also put conditions such as interfering axes clear here. For instance, if axis A and axis B should never advance at the same time, then put A Retracted in the safety rung for the five rung that moves axis B to the Advanced position, and vice-versa. This is your “sanity check” logic.

The “trigger”, or “precondition” rung is the signal that will initiate automatic motion. You can put auto conditions directly in this rung, or you can tie in auto mode sequence logic from elsewhere. The difference between the Safety rung and the Trigger rung is that the Safety rung has to stay on during the entire motion, but the Trigger run only has to be true to initiate the motion.

The Command rung is usually a sealed in coil that turns on while the machine should be trying to move this axis to this particular position. Sometimes this is synonymous with an output itself, but in many cases the Command coil will then be used to drive one or more outputs or motion instructions.

The In Position (sometimes called “complete”) rung condition is the logic that interprets the sensors to tell us that we’re actually in the given position. It’s handy to use this coil throughout your program to indicate the state of the machine, rather than looking at the inputs directly. That allows you to invert input logic or add debounce logic later in just one place, rather than everywhere that depends on this machine state.

Finally, the Fault condition is normally a timeout fault. If the command coil is on too long without seeing the In Position signal, then we determine that the motion timed out, and we stop trying. We use this fault in our alarm logic as well.

Once the command condition becomes true, it’s normally sealed in until one of three things happens: the safety condition changes to false, the motion completes and the In Position signal changes to true (normal), or the motion times out and the Fault condition changes to true.

Here’s what a generic five rung logic block looks like. I left an AFI in the trigger rung because you normally come back and fill in that logic later. Note that this is only to move one axis to one position:

The “PB” bit in the command rung is the HMI pushbutton that initiates motion to this position while you’re in manual mode. Note that both manual mode motion and auto mode motion have to follow the same safety conditions, but just have different trigger conditions.

You have to customize it for each situation. The contents of the safety and trigger rungs are obviously unique, but the command rung sometimes changes depending on the scenario. You may have more than just manual and automatic modes. Some automatic actions may actually happen when you’re not in automatic mode. Sometimes you may want an axis to “jog” (for instance a hydraulic axis moves slow, and you may want it only to move in manual mode if you hold your finger on the button) and in that case, you couldn’t seal in the Command bit around the manual mode pushbutton (you’d just seal it in around the trigger contact).

There are other variations. I hope you find this useful. If nothing else, when you come across this logic in someone else’s program, you’ll have a better idea of how it works.

plc

About 6 months ago I blogged about ControlsOverload, a knowledge-sharing Q&A site for the Industrial Automation community: Make More Money with ControlsOverload.

I was pleased to read Bill Lydon’s article in InTech called Putting knowledge to work. Bill shares a story similar to what many of us have experienced:

Early in my career, I ran up against an automation systems software problem I could not figure out. Being the “new guy” in an open office with five other engineers who had experience with these systems, I decided to get their input and explained the problem as best I could. They had suggestions, but no one offered a solution. After working on the problem for a few days, I discovered some software code that only executed under certain system circumstances, which was creating the problem. I changed the code to get the system in the field working properly and wrote an engineering change request.

I proudly shared the solution with the group. One of the most knowledgeable and experienced engineers exclaimed, “I solved that problem months ago!” Pulling out a folder from his desk file drawer, he announced his notes on the solution were, “right here.” I asked why he did not tell me this a few days before, and he responded, “You didn’t ask the right question about this specific code.” I learned this was normal operating procedures with him because he believed withholding knowledge created job security.

That’s exactly the reason why ControlsOverload was created. I’ve seen the impact of withholding knowledge, and it’s painful. Engineers are entering the industrial automation career path all the time and all of them have to re-learn the same lessons over and over because older generations aren’t taking the time to share their knowledge and mentor. It’s the opposite of the behavior that built our civilization. We need to pass these expensive lessons on to the next generation so they can spend more time building better things than we ever could.

One-on-one interaction and teaching is good, but social networking adds an economy of scale that allows knowledge sharing to explode. But somebody has to take the time to seed the field before we get our first harvest.

controlsoverload