I recently had a problem with an Allen-Bradley CompactLogix processor. The power went out, came back on, and the processor had faulted with a major fault, type 01, code 01. The fault message said “Power lost and restored in RUN mode”. There must be a way to disable this fault, and just have it go back into run mode so the operator can recover.

I Googled for the fault message and I got a link to a helpful forum thread. In that forum thread there was a link to a Rockwell Automation Knowledgebase Article that seemed to have the information I was looking for. I clicked the link and it told me I needed to login. That’s annoying enough, but fine. I tried my normal username and password for such sites, and that didn’t work. I went into my encrypted file and pulled out the username and password I’d saved for Rockwell Automation. That didn’t work.

Ok, fine. It had an option to email me my user name. I waited a few minutes for the email – yes the username I was using was OK. I clicked on the option to reset my password. Got the email, reset the password to the one I had saved anyway. Success! I was logged in, but it had forgotten what article I was trying to get to, so I went back to the original forum post and clicked the link again. “This article has been locked, or …” blah blah blah about a TechConnect ID. Ok, so I go to my profile page and click on the tab for TechConnect support IDs. None registered. Hmm, no obvious way to register one. I go to the other profile page… TechConnect ID… excellent! I enter that, click save, and now it needs my company name, address, etc. Ok, I enter that… but now I can’t click save. I have to go backwards, re-enter the TechConnect ID, and my company name and address, and then it saved. Ok, great!

I go back to the forum and click the link again. “This article has been locked, or …” What?!? I check my list of TechConnect IDs again, and there’s nothing there. This is really annoying me at this point. Oddly, under my name it now has the name of some Mexican company, and some place in Mexico. I check my profile page again and it all looks right, and I double-check the TechConnect ID. It’s right (and I know it’s right because I’ve used it to call Rockwell tech support recently).

I go back to Google and search again for the fault code, and I see a link to the Rockwell knowledge-base article. I click on that… same message! What’s going on here? How can Google even index a page that’s behind a sign-up wall and doesn’t even show you the page unless you have a valid TechConnect ID?

Going back to the original forum post, I did find some useful information in there, but obviously the knowledge-base article would have helped me the most. How far behind is Rockwell Automation’s online support? They’re still in the 90’s. I realize there are still a lot of people in this industry who are happier to pickup a phone and call their phone support in a situation like this, but as time goes on and the Millennial Generation continues entering the workforce, self-help focused people like myself are going to be more and more commonplace. We won’t settle for getting our questions answered in hours or even 30 minutes anymore; we want to solve our stupid problems like this in minutes or even seconds, and the technology is there to let us do it. Stop putting unnecessary barriers in our way! Rockwell Automation online support: FAIL!

By the way, I just spent 3 extra minutes and entered my technical question and answer over on ControlsOverload, a website for technical questions and answers about industrial automation. A website where you don’t have to sign in, and nothing is ever blocked behind any kind of wall. If Google can see it, so can you. This is the future of finding information on the internet. Rockwell Automation: Make it easy for me to use your hardware, and maybe I’ll buy more! You know what… here’s Beckhoff’s whole 350MB knowledge-base available for download so you can take it with you onsite when you don’t have an internet connection. Brilliant isn’t it? It’s called openness and it’s the new name of the game. Wake up!

Update (9 MAR 2010): The Global Quality Leader from Rockwell has contacted me and it looks like we’re going to have a constructive discussion about some of these issues (and he also gave me a different TechConnect ID to try). +1 to Rockwell for having their ears on!

controlsoverload, frustrations, plc

Recently someone asked me what the difference was between PLC programming and PC programming. This was a topic I’ve talked about before, and I gave him the same old answer I’d already come up with, but it got me thinking about it again.

Now I have a new way to compare PC programming and PLC programming: PC programming is like brick-laying and PLC programming is like sculpting.

When you write a PC program, you start out by writing the least amount of code you can to get a functional program. You run and debug that. You check it in to source control, and then you repeat the cycle, building a little more, testing it, and at regular intervals you show it to a customer, get feedback, and change direction, but you always work in small increments because it’s easier to build it that way. You make sure you’ve built a solid foundation, and then you build more on top.

On the contrary, the first time you download a PLC program to a new machine, you have a fixed number of outputs that you have to account for. You can’t just program one axis of motion first without taking into account the position of other axes on the machine (unless you enjoy the sound of metal deforming). So you start by writing logic for all the outputs and you load on more and more conditions that prevent the motion from occurring until you’re absolutely sure that nothing will move unless it’s absolutely safe. Of course, when you download this program to the machine, nothing moves, no matter what buttons you push. This is actually a good place to be. In some cases you actually put conditions in that you know could never be true, just to make sure nothing will move until you’re really sure you want it to.

In a letter from 1549, Michelangelo defined sculpture as the art of “taking away” not that of “adding on” (the process of modeling in clay), which he deemed akin to painting. (Reference)

Now that you’ve downloaded your logic to the PLC, debugging and commissioning the machine is largely a matter of taking away those conditions that you didn’t really need. This is done one motion at a time until the machine does exactly what you want, when you want, but only that. Your PLC program is done, not when there’s nothing left to add, but when there’s nothing left to take away.

That is, until you add a data collection system…

plc

(This is the third part of a trilogy of blog posts about what I think is the biggest roadblock preventing significant growth in the industrial automation industry. You should read Part 1: Industrial Automation Technology is Stagnant and Part 2: Why Automation Equipment Vendors Dabble in Integration first.)

I was recently making significant additions to a PLC program and the resulting program was too big to fit in the existing PLC. There weren’t many areas where the code could be made smaller without impacting readability, so I went out and priced the same series of PLC with more memory. I’m not going to name brands or distributors here (and I’d probably get in a lot of trouble if I did, because they don’t publish their prices), but I was taken aback by one thing: the only difference between these two controllers was the amount of memory (execution speed and features were the same), and it was over $1000 more. I realize this is “industrial” equipment, so the memory probably has increased temperature ratings, and it’s battery backed, but how much PLC memory can you buy for $1000? As of writing this post, the cost of 1 Gibabyte of memory for a desktop computer is less than $50.

At $50 per Gig, 20 Gigs of PLC memory? Nope.

Ok, it’s industrial, so… 1 Gig? Nope.

Try less than half a Megabyte. Yes.

Just for the sake of comparison, how much is that per Gigabyte? If half a Meg costs $1000, then the price of PLC memory is 2 Million dollars per Gigabyte. WTF? It’s not 40 times more expensive, it’s forty thousand times more expensive than commodity RAM.

Do you know why they charge two million dollars per gigabyte for PLC memory? Vendor lock-in. It’s that simple. They can charge thousands of dollars because the switching costs are insanely high. I can either buy a PLC in the same series from the same manufacturer (and they make sure there’s only one distributor in my area) with more memory, or I can replace the PLC with one from another manufacturer incurring the following costs:

• Rewriting the program because the languages are not portable.
• Modifying the electrical drawings significantly.
• Rewiring the electrical panel.
• Retraining all of my existing maintenance and engineering staff on the new platform.

So it really doesn’t matter. They could charge us $5000 for that upgrade and it’s still cheaper than the alternative. Why don’t they publish their prices? Simple, they price it by how much money they think you have, not by how much it costs them to produce the equipment. Wouldn’t we all like to work in an industry like that?

How did it get this way? In the PC industry, if Dell starts charging me huge markups on their equipment, I can just switch to another supplier of PC equipment. Even in the industrial PC world, many vendors offer practically equivalent industrial PCs. You can buy a 15″ panel mount touch screen PC with one to two gigs of RAM for about $3000. Add $500 and you can probably get an 80 Gig solid state hard drive in it. Compare that with the fact that small manufacturing plants are routinely paying $4500 for a 10″ touch screen HMI that barely has the processing power to run Windows CE, and caps your application file at a few dozen megabytes!

The difference between the PC platform and the PLC platform comes down to one thing: interoperability standards. The entire explosive growth of the PC industry was based on IBM creating an open standard with the first IBM PC. Here’s the landscape of personal computers before the IBM PC (pre 1980)*:

Here’s the market share shortly after introduction of the IBM PC (1980 to 1984):

… and here’s what eventually happened (post 1984):

Nearly every subsystem of a modern PC is based on a well defined standard. All motherboards and disk drives use the same power connectors, all expansion cards were based on ISA, or later PCI, and AGP standards. External devices used standard RS232, and later USB ports. Hardware that wasn’t interoperable just didn’t survive.

That first graph is exactly like where we are now in the automation industry with PLCs. None of the vendors have opened their architecture for cloning, so there’s no opportunity for the market to commoditize the products. Let’s look at this from the point of view of a PLC manufacturer for a moment. Opening up your platform to cloning seems like a really risky move. Wouldn’t all of your customers move to cheaper clones? Wouldn’t it drive down the price of your equipment? Yes, those things would happen.

But look at what else would happen:

• The hardware cost of your platform would drop, so everyone would leave your competitor’s platforms and come to yours. At least you’d still be in the game, but your competitors wouldn’t even have a compatible platform anymore.
• By dropping the price of automation equipment, the demand for complementary products would increase. IBM didn’t make its fortune directly in the PC industry, but they rode the wave and became an enterprise services company built on top of the commodity PC business.
• The size of the industry would explode.

Why don’t we have open standards? What about IEC-61131-3? PLC vendors tell us that their controllers are IEC 61131-3 compliant, but that doesn’t mean they’re compatible. The standard only specifies what languages must be supported and what basic features those languages should have. Why didn’t the committee at least specify a common file format? You may be surprised to learn that many of the major committee members were actually from the PLC equipment manufacturers themselves. The same people who have a vested interest in maintaining their own vendor lock-in were the ones designing the standard. No wonder!

What about the new XML standard? Well, it’s being specified by the same group and if you read their documentation, you can clearly see that this XML standard is not meant for porting programs between PLC brands, but rather to integrate the PLC programming tools with 3rd party programs like debugging, visualization, and simulation tools. You can be certain that a common vendor-neutral file format will not be the result of this venture.

Solutions

I see three ways this problem could be solved:

1. An association of common manufacturers and system integrators could form and develop a truly open and interoperable standard,
2. A PLC manufacturer could “go IBM” and open up their platform, or
3. A single integrator or manufacturer could develop and create an open automation platform and put the rights in the public domain.

Option 1 is almost doomed to fail because it has the word committee in it, but options 2 and 3 (which are similar) have happened before in other industries and stand a good chance of success here.

In my opinion, the most likely candidate to fulfill option 2 is Beckhoff. I say this because they’ve already opened up their EtherCAT fieldbus technology as an open standard, and their entire strategy is based around leveraging existing commodity hardware. Their PLC is actually a real-time runtime on a PC based system, and their EtherCAT I/O uses commodity Ethernet chipsets. All they would really need to do is open source their TwinCAT I/O and TwinCAT PLC software. Their loss of software license revenue could be balanced by increased demand for their advanced software libraries in the short term, and increased demand for their EtherCAT industrial I/O in the longer term. Since none of the other automation vendors have a good EtherCAT offering, this could launch Beckhoff into the worldwide lead relatively quickly.

For option 3, any company that considers automation equipment to be an expense or a complementary product (i.e. manufacturing plants and system integrators) could do this. There is a long term ROI here by driving down the cost of automation equipment. Many internet companies do this on a daily basis. IBM sells servers and enterprise services, which is the natural complement of operating systems, so they invest heavily in Linux development (not to drive down the cost of Linux – it’s free – but to offer a competitive alternative to Microsoft to keep Microsoft’s software costs down). Google does everything it can to increase internet usage by the public, so it invests heavily in free services like Google Maps, Gmail, and even the Firefox web browser. The more people who surf, the more people who see Google’s advertising, and the more money they make.

If I were going to go about it, I’d build it like this:

• Base it on commodity PC hardware (x86).
• Pick an open source, free real time operating system like FreeRTOS.
• Document and publish (free download) the communication protocol and file format for the automation program files (e.g. ladder logic).
• Write a reference runtime that conforms to this communication protocol and publish it under a BSD license. It doesn’t have to be great, but it has to work and be useful.
• Write an extensible programming environment (Windows-based) where you can develop automation programs that conform to the standard and that communicates with any runtime that conforms to the standard. Give it away free and publish it under the GNU GPL license to prevent it from being embraced and extended in a proprietary way by one vendor.

If you did that, anyone could make hardware that’s compatible with this platform, people are free to innovate on the runtime, and anyone can make improvements to the programming environment (as long as they give their changes back to the community). Instant de-facto standard.

I know some of you may be grinning right now because this is the path I’m following with my next personal software project, that I’m calling “Snap” (which stands for “Snap is Not A PLC”). However, Snap is for the home automation industry, not the industrial automation industry. If any company out there wants to take up the gauntlet and do this for industrial automation equipment, you may just change this industry and solve our biggest problem.

Discussion is welcome. Please comment below or contact me by email (e8pzrhs02@sneakemail.com).

* Graphs courtesy of ars technica.

innovation, plc, standards

Have you ever wondered what the ladder logic for a dog would look like?  I imagine it’s something like this:

… and so on.  I think that’s why we consider dogs so loyal.  Perhaps by loyal, we mean predictable, or understandable.

Ever wonder what the ladder logic for a cat would look like?  I imagine it thus:

… or something like that.  Actually I’m pretty sure I’ve met a couple of cats that came equipped with thirteen sequencers and a conditional subroutine jump in there somewhere.  It certainly makes life interesting.  Does that little tail wag mean it’s safe to pet, or does it mean your cat is about to mistake your inner thigh for prey?  Who knows!  What fun!

I guess my point is, cats can be moody, and believe it or not, so can machines.  You might call it “internal machine state”, but I call it moodiness.  Have you ever been trying to troubleshoot a machine and it was stuck thinking there was a part in one of the stations that really wasn’t there?  Every time it indexes it keeps faulting?  That’s machine moodiness.  So there you are, flagging every sensor in sight trying to get that part present bit to clear, and no matter how many roses or chocolates you buy for the darned thing, you know you’ll be sleeping in the dog house tonight.

Thankfully, there’s a cure for machine moodiness:  Make all internal state visible and editable.  At the very least, there should be a screen on the HMI that shows the current status of the part present bits at each station.  If you really want to be fancy, make sure it allows the operator to set and clear those bits manually.  That includes latches, sealed coils, counters, FIFOs, and even long running timers.

Anyway, if you can’t see it, you can’t troubleshoot it, so adding visibility will save you time in the future.  Trust me.  And trust dogs; they’re quite loyal.

internal-state, ladder-logic, plc

“Why do you engineers insist on using sequencers in your PLC programs?”

Mike asked that question the last time I was over at the local washing machine manufacturer.  Mike works in the maintenance department there.  As usual, a five minute job had turned into over two hours of frustration, and I just happened to be the sympathetic shoulder.

First, let me disclose a little known secret about how washing machines work.  You see, early in the days of washing machines, certain garments, usually socks, ended up getting wrapped around the spindle causing a jam, and this eventually causes the motor to overheat and burn out.  Frustrated by this problem for over a decade, manufacturers finally solved this problem by incorporating a device known affectionately in the industry as a “sock shredder”.  When wayward socks start to get tangled around the spindle, the extra torque on the motor activates a torsion bar that engages a spinning cutting head that pulverizes the sock into fine enough fibers that most of it simply gets carried away in the rinse cycle.  Even though companies were worried about the backlash from customers, the initial consumer trials showed that most people were rather ambivalent about the missing socks, so the device is now incorporated into every washing machine on the market today.

(Not only did the laundry industry benefit from this innovation, but sock manufacturers reported a 62% rise in demand over a 5 year period, unprecedented in the undergarments industry both before and since.)

Anyway, back to Mike and his frustration.  Apparently the plant had received complaints that a few washing machines were experiencing premature failure of the motor.  Apparently the machines had left the plant without the sock shredders installed.  Mike explained to his boss that it would be easy to install a photoeye that checked for the presence of the sock shredder at the spindle insert station, and stop the inserter if the sock shredder wasn’t detected.

After wiring in the photoeye, Mike went online with the PLC, started looking for the output that actuated the inserter cylinder but was surprised to discover it wasn’t referenced as a coil.  As you can probably guess by now, the outputs were being driven by a sequencer instruction.

Ancient History

PLCs first appeared in the 60’s.  While most historians believe that PLCs were intelligently designed, a few skeptics maintain that they evolved from less advanced technologies.  But I digress…

Back in prehistoric times (before PLCs), machines used mechanical drum sequencers (just like the valve timing in older automobiles) to sequence the actions of the machine.  The drum triggered electrical cam switches that would turn on outputs in the correct sequence to operate the machine.  When PLCs first hit the market, the PLC designers thought it would be a good idea to create a “sequencer” instruction that would mimic the operation of the drum sequencer to make it easy for machine manufacturers to convert their existing machines over to PLCs.  That was decades ago.  So, why are engineers still using sequencers?

Machine builders use sequencers because they’re building a machine with a standard operation and it’s easy to go straight from a timing diagram to a sequencer.  (Some programmers will tell you that sequencers use less PLC memory, but with the price of memory these days, that’s not a valid excuse.)  Of course, when the machine hits the plant floor and needs to be modified, the presence of sequencer instructions frequently triggers severe hair loss in the maintenance department.

Solution

Sequencers can easily be replaced with logic made up of contacts and coils.  Each step can be represented by a single coil that is sealed in when the step is complete.  I find it convenient to also use a second coil for each step called “step in progress”, which is equivalent to “previous step complete and current step not complete”.

Use the “step in progress” coils to drive the outputs directly.  The logic goes from obfuscated to obvious.

Steps don’t have to be numbered either.  With tag based PLC processors you can create steps called Lather, Rinse, and Repeat.  Then if you need to add a new step called Scrub between Lather and Rinse, there’s no awkward shuffling of step numbers.

Bottom Line

For Mike’s sake, don’t use sequencers!

References

• http://www.eod.gvsu.edu/~jackh/books/plcs/chapters/plc_adv_func.rtf
• http://www.plcs.net/chapters/history2.htm

plc, sequencer