I got the OWI-535 “Robot Arm Edge” 5-Axis robot arm and USB Controller add-on for Christmas:

The robot arm comes as a kit that you assemble yourself, and my 3 year old and I had lots of fun putting it together (it helped to have some tiny fingers around, honestly). It comes with a manual controller that allows you to rotate all 4 joints, plus the gripper. It’s fun to play around with, but let’s be honest, everyone wants to hook it up to their computer.

Unfortunately the software that comes with the USB controller works on Windows 7, but “32-bit only”. That was a little annoying, but hey, I didn’t really want to stick with their canned software anyway. I figured I would see if I could get it to work from C#. I was surprised to find a great site by Dr. Andrew Davison in Thailand who wrote some Java code as part of his site for his book Killer Game Programming in Java (chapter NUI-6 Controlling a Robot Arm, which doesn’t appear in the book). Surprisingly he went as far as to do the inverse kinematic equations so you can give the arm a set of X,Y,Z co-ordinates (in mm) in “world frame” and it will calculate all the join angles to get to that location, and then used timed moves to get the arm into that position.

His Java library uses libusb-win32, and that library has a .NET equivalent called LibUsbDotNet. The API’s don’t appear to be the same, but thankfully I managed to find a thread on SourceForge about controlling the OWI-535 using LibUsbDotNet. So, over the course of a couple of nights, after the kids went to bed, I ported Dr. Davison’s Java code over to C# (quite easy actually) and replaced the libusb-win32 calls with LibUsbDotNet API calls. It worked!

Here is the .NET solution that I wrote called TestOwi535. I used Visual C# 2010 Express Edition to write it, so that’s free. Also, you must download and install LibUsbDotNet first and run the USB InfWizard first to generate a .inf file (you have to do this with the robot arm plugged in and turned on), and then use that .inf file to install LibUsbDotNet as the driver (technically you’re installing libusb-win32 as the driver and LibUsbDotNet is just the C# wrapper).

If you right click on the C# project in the solution explorer, you’ll find 3 options for startup object: MainClass is the original code I found in the SourceForge thread, but it’s just proof of concept code and only moves one joint in one direction for a second. The ArmCommunicator class is an interactive console application that allows you to move all joints (and control the gripper and light) by typing in keyboard commands. Finally the RobotArm class is the full inverse kinematics thing. In the last case, make sure you start with the arm at the zero position (base pointing away from the battery compartment, and all joints in-line straight up in the air, gripper pointing directly up, and gripper open). It will do a move to the table, to the front right of the arm, close the gripper, and then move back to zero and open the gripper.

Unfortunately that’s where you start to see the obvious deficiency of the arm: it has no position feedback. That means even though it tracks its position in the code, the physical position eventually drifts away from the internal position tracking, and the arm eventually doesn’t know where it is (it’s just using timed moves). One of the biggest areas where you could improve it is to change the joint rates so that it knows that the joints move faster when going down than when going up.

Still, it’s a neat little toy for the price. I’m going to start hunting around for a way to add joint position feedback, as that would really improve the performance. I also want to rewrite a new module from the ground up that allows concurrent joint moves (this one only moves one joint at a time). Ideally you want to control this thing in “gripper frame” instead of “world frame”.

Happy hacking!

It’s easy to overlook the power of human motivation in automation systems.

I’m going to assume you don’t work in a lights-out factory. That’s pretty rare. Almost all automation systems interact with people on a regular basis, and even though we have high fidelity control over our automation processes, people are notoriously difficult to predict, let alone control.

For instance, consider a process with a reject station. Finished parts are measured and parts that don’t meet specification are diverted down a chute. Good parts continue down the line.

Question: how big do you make the reject bin? Naively you might want to make it as big as possible so the operator doesn’t have to waste time emptying it. Unfortunately that means it’s just as easy to make bad parts as it is to make good parts. You’d be better off to make the reject chute only hold 3 parts, and put a full sensor on the chute that throws a fault and stops the machine when it’s fully. Then it’ll be a pain in the ass to make bad parts, and the operator will have a lot more motivation to do something about it. While you’re at it, put the reject chute on the other side of the machine so they have to walk around the machine to empty it.

Consider a machine with an e-stop button. It’s a big red button with a mushroom head that’s supposed to be easy to hit. However, I’ve seen a lot of machines where the consequence of hitting that button was major downtime because the part tracking got screwed up or the machine just didn’t recover gracefully. I once watched a pallet get bumped out of the track and it was riding along the rail of the conveyor. I hit the e-stop just before it was about to damage some equipment. I was scolded for my efforts: “never hit the e-stop,” he said. That’s the wrong motivation. You want operators to press the button when they see something wrong, so make it easy to recover.

Consider an inventory tracking system. You want people to record stuff they’ve consumed, and what cost account they’ve consumed it against. What motivation does a person standing there with a bolt in their hand have to look up that bolt in your inventory system and mark it consumed? Very little. What if you lock the door to the store room and make them request an item before you unlock the door? That’ll help, but chances are some people who don’t know exactly what they want will click the first item on the list, and just go in and browse. What if you make the inventory storage system so convoluted that the only way to find an item is to look up the storage location in the computer? Well, that might work (until your inventory system breaks).

Like water, people tend to take the easiest path down hill. You’re better off digging a channel where you want it to go than expecting it to get there under its own power. Use gravity to your advantage. Make it harder to do things the wrong way and easier to do things the right way.

There’s been a lot of discussion recently about the dangers of connecting automation equipment to networks, and yet there are significant pressures to do so. Of course, I don’t ever think that you should take a PLC and put it on an internet accessible IP address, but it’s certainly common practice to connect industrial automation equipment to internal LANs to facilitate data collection. People in the front office need to push production planning information down to the production floor, and they need real-time data on what’s going on (not to mention for historical data logging, historians, etc.).

It’s all too common to throw a PLC on the same network as your front office, and I’ve seen it blow up. What happens is something invariably goes wrong on the office network (someone plugs two ports together on the switch in the boardroom, or someone brings in an infected music player and plugs it in, or the DNS server at head-office goes down and the local DNS doesn’t work correctly… I’ve seen a lot). However, you want your machines to keep going when this happens.

This is all made worse now that (a) industrial automation equipment is more commonly based on off-the-shelf commodity hardware and software (e.g. windows PCs) and (b) the people writing malware can actually spell PLC now. Up to this point there’s been some form of security through obscurity.

If you have a local IT staff that’s on the ball, you really should be getting them to handle the network layout. On the other hand, if you’re in a small facility with limited resources, there’s a lot you can do by making some simple design choices that will go a long way towards improving the reliability and security of your systems.

Most automation cells now come with an Ethernet switch already built-in. Typically this is an industrial spec. DIN-rail mounted one. It’s not fancy, but it’s supposed to survive in a panel. These Ethernet switches are there to connect your PLC to your HMI, and increasingly to connect your PLC to Ethernet-based I/O like Ethernet/IP, etc. The common (and wrong) thing to do is to drop a network cable from your plant network to your panel and plug it right into this Ethernet switch. This creates some technical problems right off the bat:

The automation devices typically have fixed IP addresses (I personally prefer this because it means these devices aren’t dependent upon an external DNS or DHCP server – two less dependencies are good). Chances are that these IP addresses won’t work on your plant network, so you have to manage those IPs at the plant level. You’re opening yourself up to someone with the wrong IP on their laptop pouncing on your PLC’s IP address, and then bam, your machine is down.

A much better way is to place some kind of Router with NAT between the plant network and the machine’s Ethernet Switch:

Now if you’re just a little manufacturer with two machines out back and your data-collection link isn’t critical, you can probably get away with one of those home routers from Best Buy that you’d use to connect your laptop and your desktop at home to your cable modem. Note that it doesn’t need to be wireless, and you’re probably better off if it isn’t. The way you hook it up is to connect the Internet (Uplink) port on the router to the plant network and run a cable from one of the ports on the LAN side to the existing Ethernet switch in the machine. If your data needs to be a bit more reliable, consider buying some kind of Cisco router with NAT capability (but you’ll be going from the $50 range to many hundreds of dollars – your choice).

Now, when you configure it, you want to make sure that you turn off the port forwarding function, the DMZ function, disallow remote administration, and block all anonymous internet traffic (these should be default settings, but it’s good to check). Also, make sure the router’s local IP address doesn’t conflict with the PLC’s and HMI’s, and make sure they have the same subnet. Typically you’ll want to either turn off DHCP on the local side, or limit it to a range that won’t conflict with the fixed IPs. DHCP is handy when you connect your laptop to the programming port. Now what you’ve done is made it somewhat invisible from the plant side. Some piece of malware scanning for devices on your plant network should just see a black hole.

Now on the PLC side, you can now initiate a connection to the data collection server even though the data collection server can’t connect to the PLC (in the same way that your home computer can connect to Google, but Google can’t get to your PC – theoretically). Note that a piece of malware on your data collection server or on one of the routers/switches in your plant network could intercept this communication, and own your PLC, but at least you’ve significantly reduced the surface area of attack. Not perfect, but reasonable at this time, depending on the sensitivity of your equipment. I’m assuming you’re not enriching uranium or providing drinking water to my community.

(I’m going to be talking specifically about Allen-Bradley products now – sorry.)

So how do you get the data from the PLC to the data collection server? In the old days you’d have some software on the server like RSSQL, and it used a product like RSLinx Enterprise and as far as I know, it initiated the connection to the PLC. That won’t work in this case. Sometimes you’d throw an OPC server in there, and have some kind of historian that would log tags to a database. That OPC server, obviously, needs to be able to initiate a connection to the PLC. To use a router with NAT, you’d need to port-forward from the router to the PLC (or to the OPC server if it was inside the machine network). That’s undoing a lot of our protection.

What you need to do is initiate the connection from the PLC, and have the Data Collection computer act as a Server. One way to do this is with a 3rd party Ethernet card, like this MVI56-GEC card from Prosoft for the ControlLogix line. I have used that in the past to connect to a server, but it involves a lot of ugly PLC coding. It’s your only option if you have to conform to someone else’s protocol though.

If you just want to write data directly into a SQL database, there are 3rd party products that will let you do this (basically a SQL Server connector card).

But there is an option without buying any new hardware. The ControlLogix/CompactLogix lines can send Unsolicited CIP messages, and you can find products that can receive these messages in the PC world, like CimQuest’s NET.LOGIX product. It can act as a server and receive data directly from the PLC – either individual tags, or even arrays of UDTs. The code on both ends is relatively simple, so all you have to pay for is the NET.LOGIX runtime license, which is cheaper than the hardware alternatives. Note that you can also do this with PLC5 and SLC500 devices, though there’s some more effort involved.

I hope that’s enlightening. This is by no means a perfect solution, but it’s reasonable for now. It doesn’t plug the laptop hole (the programming laptop is probably still your #1 vector for malware to get into your machine network). It’s susceptible to man-in-the-middle attacks between the router and the data collection server. It’s susceptible to exploitable bugs in the router’s firmware. Beware and use your own judgement.

Completing the trilogy of ICS-security related blog posts, a hacker recently demonstrated how easy it was to find and log in to an internet-facing SCADA system using for water management in a town in Texas. From the article on threatpost:

The hacker, using the handle “pr0f” took credit for a remote compromise of supervisory control and data acquisition (SCADA) systems used by South Houston, a community in Harris County, Texas. Communicating from an e-mail address tied to a Romanian domain, the hacker told Threatpost that he discovered the vulnerable system using a scanner that looks for the online fingerprints of SCADA systems. He said South Houston had an instance of the Siemens Simatic human machine interface (HMI) software that was accessible from the Internet and that was protected with an easy-to-hack, three character password.

For those of us who design, build, and deploy systems like this, let’s ask ourselves what would happen if a serious incident happened and significant equipment damage was done, or worst case, people were seriously injured or killed. Don’t you think the people who worked on the system would end up in court (if not in criminal court, then at least in civil court)?

When in doubt, don’t sit these things directly on the internet. There are lots of secure remote access products available (Google for “VPN”). It’s worth it.

Joe Weiss recently reported on the possible hacking of a public water SCADA system, apparently in Illinois. This attack, if it was an attack, caused damage to a pump by turning it on and off repeatedly.

It seems obvious that this situation is going to be repeating itself more and more. If you’re a company with industrial control systems, or you provide control system services, now’s a great time to start thinking about your control system security strategy. Do you have the necessary skills on staff? If not, where are you going to source them from?

I think most people in our industry realize you shouldn’t connect industrial automation devices to the internet, but just in case you happen to think otherwise, here’s a quick explanation why (this is old news, by the way).

You may believe that things connected to the internet are relatively anonymous. There’s no web page linking to them, so how is Google going to find them, right?

It turns out it’s relatively easy to find devices connected to the internet, and it’s kind of like the old movie WarGames where the lead character, played by Matthew Broderick, programmed his computer to dial every phone number in a specific block (555-0001, 555-0002, etc.) and record any where a modem answered. That was called “war-dialing”. In the age of the internet, you just start connecting to common port numbers (web servers are on port 80, etc.) on one IP address at a time, and logging what you find. This is called port-scanning.

It turns out that you don’t even have to do this yourself. One free service called SHODAN does this for you, and records everything it finds at the common port numbers (web servers, FTP servers, SSH daemons, etc.) and lets you search it just like Google. It turns out that (a) most modern industrial equipment is including embedded web servers and/or FTP servers to allow remote maintenance, and (b) most web servers or FTP servers respond with some kind of unique “banner” when you connect to them, announcing who or what they are.

So, if you don’t believe that you shouldn’t be putting industrial automation equipment on the internet, here’s a little experiment you can run:

1. Take a ControlLogix with an ENBT card and hook it directly to the internet, so it has a real IP address.
2. Wait a couple of days.
3. See if your IP address shows up on this SHODAN search page.

You could try the same thing with a Modicon M340.

This query for Phoenix Contact devices is particularly scary, as one of the links is a wind turbine! I was a bit scared once I opened it (it opens a publicly accessible Java applet that’s updating all the data in real-time), so I closed it. There was no password or anything required to open the page. At least the button that says “PLC Config.” appeared to be grayed out. Let’s hope that means it’s protected by some kind of password… and that it’s hardened better than every single major corporation’s website was this year.

Just want to say thanks to DigitalBond for pointing out this SHODAN search for all Advantech/Broadwin WebAccess deployments around the world too.

It’s fascinating to watch what’s unfolding in the Industrial Control System Security front these days. Digital Bond’s SCADA Security Portal is as entertaining as any (thanks to ArchestrAnaut for pointing it out for me).

A brief recap:

• Stuxnet makes news even in the mainstream press
• Siemens shrugs it off and does absolutely nothing about it
• Security researchers, smelling smoke, start poking around PLC security and find it completely lacking
• Details about wide open backdoors inserted into common PLC hardware has now been published online

Things are not moving in a positive direction either. Those security “researchers”, many of whom seem to be selling security solutions, are digging up ways to compromise PLCs and they’re posting all that information online. Now if this forces automation vendors to stop looking the other way and start taking security seriously, then I think it can only be a good move in the long term, but you have to admit it feels a little like a tire salesman throwing roofing nails on the road in front of his store.

All of this makes you wonder, what’s a small manufacturer to do? As always, businesses need to weigh the risks and the costs and act accordingly. This isn’t easy for the decision makers. On one side there’s enormous pressure to network all of the systems together to facilitate the fast flow of information between the ERP, MES, and Plant Floor layers, but on the other side, every interconnection increases the risk of catastrophic failure. I’ve personally seen Windows worms take down automation networks. In the next few years I’m certain we’re going to see worms that can jump from PLC to PLC and probably ones that can cross from Windows to PLC and back.

Properly segregating networks and then managing them is a big IT project. That means it needs scarce resources, and those resources aren’t making money for the company. Big manufacturers have enough cash flow (and have been bitten enough times) that they can allocate resources for this kind of project, but small manufacturers are a different story.

Small companies generally lack the specialists needed to implement such systems. Almost by definition, generalists serve in small companies and specialists gravitate towards large companies. Small companies can only implement commodity solutions (unless it’s part of their core business strength). That means that while we’re all worried about what might happen if a major utility or top tier manufacturer gets hit with an automation security breach, the fact is it’s more likely that small manufacturers will be the first ones hit by a fast-spreading generalized threat. The economic impact could be just as large… those small manufacturers are feeding parts up the supply chain, and in this just-in-time environment it doesn’t take much to cause a major interruption.

What’s the solution?

Short of the automation vendors waking up and making secure products, we need better (and less expensive) tools for securely connecting our PLCs. I hate to say it, but you can’t implement modern control systems without knowing the basics of network security, VLANs, and access control.

A little while ago I told you about an ongoing project to automate my parents’ G-Scale outdoor model railroad. Today I’m going to add a bit more detail about the solution: specifically, how do you sense the location of the trains?

Layout

The layout is broken into 3 lines: a “Figure 8″ line, a “Point-to-Point” line, and a “Main Line”, which has various track switches, sidings, etc. The interesting thing about the Figure 8 and Point-to-Point lines is that they cross, and one of the goals is to prevent trains from colliding.

Some other goals include:

• The point-to-point should start at one station, move to the other station, stopping out of site for a while. It should stop at each end-station for an adjustable period of time, and return.
• The Point-to-Point and Figure 8 lines have uphill and downhill sections, so the speed needs to be varied in these sections.
• The Figure 8 line has two programmable stops, but the train shouldn’t necessarily stop at both stops every time.
• We want to run multiple trains on the main line. Some will be stopped on a siding while others are running, and then they will switch.
• All tracks need the ability to be manually operated.

The Figure 8 Line from behind "Stan's Speed Shop"

Power

All existing locomotives in the system use “track power” (DC voltage applied across the two rails of the track). The voltage applied to the track is applied to the motors in the locomotives, and this controls the speed.

There are some advantages to this: it allows you to run “stock” (unmodified) engines, and it’s compatible if someone wants to bring over a “guest” engine (either a track powered or battery powered model). It’s also compatible with “DCC” controlled locomotives which, as I understand it, are backwards compatible with track powered systems.

Control?

Whether you use a PLC or a PC for control, being able to control the voltage to a track (to control a motor speed) is pretty much a solved problem, so let’s assume we can do that for now. The main problem is location sensing. In order to tell the locomotives to stop, wait, start, etc., you need to know where they are.

Your first thought, as a controls engineer, is some kind of proximity sensor. Unfortunately there are some significant problems with this:

• Metal-sensing proxes are expensive, and the locomotives are mostly plastic. We’re trying to avoid retrofitting all locomotives here. You might be able to sense the wheels.
• Photo-sensitive (infrared) detectors, either retro-reflective or thru-beam type, are popular on indoor layouts, but they apparently don’t work well outdoors because sunlight floods them with infrared.
• Reed switches are popular, but you need to fit all your engines with magnets, and they are a bit flaky. Magnets and reed switches actually work well if you have the magnet on the track, and the reed switch mounted to the engine, in order to trigger whistles, etc., but even then they’re not 100% reliable, in our experience.
• All proximity detection strategies require you to run two wires to every sensor, which is a lot of extra wiring. Remember, there are lots of little critters running around these layouts, and they tend to gnaw on wires. Fewer wires is better.
• Having sensors out in the layout itself means you’re exposing electrical equipment to an outdoor environment. At least you can take the locomotives in at night, but the sensors have to live out there year-long. I’m a bit concerned by that thought.

Solution: Block Occupancy Detection

The solution we found was a technique called “block occupancy detection.” This is a fairly common method of detection in some layouts. A couple of years ago, I built a simple controller that solved the crossing detection problem between the Figure 8 and Point-to-Point lines using block detection to know where the trains were. It worked great, so we decided to use it for the entire system.

Here’s how it works: you divide up your track layout into “blocks”. Blocks can be any size, but they are typically anywhere from about 4 feet long, up to the length of a train, or a bit more. One rail on the line is “common” and isn’t broken up. The other rail is the one you cut into electrically isolated sections.

So, the wire from the common side of your speed controller goes directly to the common rail, as it did before. However, you have to split the “hot” side of your speed controller into as many circuits as you have blocks. Each block is fed from a separate circuit, which means you have to run a “home run” wire from each block back to your power supply.

Then, the “block occupancy detection” circuit is wired in series with each block circuit (between the speed controller and the block). Here’s what one block detection circuit looks like:

This is an interesting circuit. On the left you can see a bridge rectifier, with the + and – terminals curiously shorted out. This is a hacked use of the device. All we really care about is that we want to create a voltage drop across the device when current is flowing through the wire to that block. One diode creates a voltage drop of 0.6 to 0.7 V, and the way we’ve wired it, whether the speed controller is in forward or reverse, the current always has to take a path through two forward-biased diodes. That means, when current is flowing to the block (i.e. there’s an engine in that block) then we get a voltage drop of 1.2 to 1.4 V across this device (or -1.2 to -1.4 V if it’s in reverse). A standard bridge rectifier is just a handy component to use because it’s readily available in high current ratings for a couple of dollars each.

We’re using that constant voltage to drive the input side of an LTV-824 opto-isolator chip. Notice that it’s a bi-directional opto-isolator, so it works in forward or reverse too. On the output side of the opto-isolator, we can run that directly into a PLC input (the input we’re working with here is sourcing and has a pull-up resistor built-in).

If you’re using a regular straight-DC analog controller, that’s all you need, but in this case we’re using a pulse-width-modulated (PWM) speed controller. That means the PLC input is actually pulsing on and off many times a second, and if you’re at a slow speed (low duty cycle), then the PLC may not pick up the signal during it’s I/O scan. For that reason, I found that sticking a 1 uF capacitor across the output will hold the PLC input voltage low long enough for it to be detected quite reliably. This, of course, depends on your pull-up resistor, so a little bigger capacitor might work better too.

Filtering in the PLC

This worked quite well, but needed a bit of filtering of the signal in the PLC. The input isn’t always on 100% while the locomotive is in the block, so once a block is latched as “occupied”, I use a 1-second timer of not seeing the input on before I decide that the block is clear.

I also have to see an adjacent block occupied before I clear a block. That solves the problem of “remembering” where an engine is when it stops on the track and there’s no longer any current flowing to that block.

Of course, this means you can end up with “ghosts” (occupied blocks that are no longer truly occupied because someone picked up a locomotive and physically moved it). I provided some “ghost-buster” screens where you can go in and manually clear occupied blocks in that case.

Pros and Cons

I like this solution for several reasons: all the electronics are at the control panel, not out in the field (except the wires to each block, and the track itself). Also, the components for one block detector are relatively inexpensive, and we’re working on a bit of a budget here (it is a hobby, after all). I think reliability and simplicity also fall into the Pro column. As long as you can get a locomotive to move on a segment of track (the track needs cleaning from time to time), then the PLC should be able to detect it. You don’t need to deal with dirty photo-detectors or extra sensor wires. The same wire that carries the current to the block carries the signal that the block is occupied.

On the other hand, there are some negatives. This system, as designed, has 21 different blocks, which means 21 home-run wires, buried in the ground, in addition to the commons (plus the track switch wires, but that’s a story for another post). Also on the negative side, you don’t get 100% accurate position sensing. Actually, you get pretty accurate sensing at the edges of the blocks (you’re pretty sure you know where the locomotive is the moment is crosses from one block to the next), but you’re not sure where it is in the middle of the block.

You do have to make other compromises in the track system. There are some accessories (like lighted end-stops) that draw power from the track. This current draw makes that particular block show up as occupied all the time. You either have to modify the accessory to use battery power, or you have to run extra wires to that accessory.

You also have to take the length of train into account. You know which blocks are occupied by current-drawing locomotives and cars (like lighted observation cars and caboose’s), but not every car draws power. Your design and control system needs to take into account whether or not your train will occupy more than one block at once, and whether the end of the train will be detected. This is most important when trying to run multiple trains on one track, where you want the back train to avoid running into the end of the first train.

Next

I hope that’s been educational. I’m still not done programming the PLC, and I’m waiting for a component to arrive for the throttle controller right now. I’ll post more information over the next few weeks.

In programming, we have a principle called Don’t Repeat Yourself (DRY). It’s a very important idea, and I’d argue that most of the advances in programming environments over the years have been in support of this principle and its related principle, Once and Only Once (OAOO).

Unfortunately, like every “principle”, it eventually takes on the level of dogma, and the people spouting it sometimes forget why it exists. These principles aren’t ends in themselves; they’re not self-justified. They are general principles to follow, but only when they support the end-goal of solving problems in more efficient, and more maintainable ways.

Let me give you a very simplified example of how it can be carried to far. Consider the following declarations in C#:

const int MOTOR_1_START_TIMEOUT_MS = 5000;
const int MOTOR_2_START_TIMEOUT_MS = 5000;

Consider that I could write:

const int MOTOR_1_START_TIMEOUT_MS = 5000;
const int MOTOR_2_START_TIMEOUT_MS = MOTOR_1_START_TIMEOUT_MS;

or…

const int MASTER_MOTOR_TIMEOUT_MS = 5000;
const int MOTOR_1_START_TIMEOUT_MS = MASTER_MOTOR_TIMEOUT_MS;
const int MOTOR_2_START_TIMEOUT_MS = MASTER_MOTOR_TIMEOUT_MS;

Notice that all 3 versions accomplish the same end-result, but they are semantically different. The first version means that the two motors have independent timeout values, and they’re just co-incidentally the same. The second says, “motor 2′s timeout must be the same as motor 1′s timeout.” The third says that both motors must have the same timeout.

In my opinion, any of these three versions might be correct for various systems involving two motors. However, if you follow the DRY principle without thinking about it, you’ll assert that the first version is incorrect. In fact they’d probably say the only correct version should be:

const int MOTOR_TIMEOUT_MS = 5000;

(…ignoring, for the moment, that it should probably be a configurable value rather than a constant.)

Why does this simple example matter? Consider the case of a PLC-based control system with 10 motors. Let’s say at the start that all the motors, and all the drives running them, are identical. If you’re familiar with my philosophy of PLC programming, you know that my default solution for this would be to have 10 ladder logic routines, each called MOTOR_01, MOTOR_02, etc. Each routine would basically be a copy. That really doesn’t follow the DRY principle, does it? Certainly no, not at face value.

You might not believe it, but I get the occasional “hate mail” to my blog’s email address because of some of my technical opinions here. The most recent one, comically, referred to me (and all PLC programmers for that matter) as “dinosaurs”. I’m not sure what the rest of the message said, because if you can’t be polite, I’m not going to bother listening to you. However, I believe it’s this flagrant violation of things like the DRY principle that really rubs traditional PC programmers the wrong way when you start to talk about the principles of PLC programming.

Of course, my views about PLC programming are just that – general principles that need to be evaluated in the light of each and every project. I’m just asserting that most of the time you should be following a principle of a one-to-one mapping between ladder logic and real-world hardware. That doesn’t mean it’s an unbreakable rule.

Going back to the 10 motor example, the way you structure your program should be based on a decision you make about anticipated future changes to the system.

If you write one generic routine for controlling a motor, and you call it 10 times, you’re saying, “I always expect all 10 of these motors to behave in an identical way for all of the future.” Of course, you can allow variations, but you have to do that by passing in parameters for each instance. You have to be explicit about what can vary. Adding new parameters is typically a harder task than just modifying one of the 10 existing motor routines when you need to change the behavior of one motor.

On the other hand, if you follow my principle of 10 motor routines for 10 motors, you’re saying, “I expect that we’ll rarely need to make a sweeping change to all 10 motor control routines, but that we are likely to modify one or two routines to make them perform differently than the others.” I personally believe this is usually closer to the truth. As a system ages, perhaps one motor drive will blow, and you can’t buy the original drive anymore, so you have to replace it with a new one that has different control signals. That’s a fairly typical scenario, in my experience. Also, even though you might have 10 identical drives and motors, the process may or may not be identical for each motor. They may perform vastly different functions, and it’s likely that you’ll want to change just one or two of them to access more advanced features of the drive when you refine the process. Of course, I also like that with a one-to-one mapping in a PLC, troubleshooting becomes much easier because with online monitoring you can see each control routine executing just for that motor. You can make temporary changes just to one motor routine to bypass a faulted drive, or to do a million other changes that you’ll never be able to predict when you’re writing the logic.

The fact is, we’re physically limited by the number of drives we have. The amount of time it takes to make a change to all 10 motor control routines is tiny compared to how long it takes to make physical changes to 10 drives. This effort scales with the size of the system. In PC programming, you can have a system with millions, even billions, of objects, but in the PLC world, you’re limited by physical reality. The consequences of repeating yourself aren’t always as great, and you need to take that into account, and weigh it against your other goals.

That doesn’t mean I can’t imagine a case where you really want to assert that the motors all have to operate identically, all of the time, forever in the future. There are systems with load sharing drives where the system wouldn’t operate if you mismatched the drives or motors. That’s a design decision you have to make. Principles are only there for guidance, but they are not absolute rules, and they shouldn’t be treated that way.

Several years ago my parents decided they were going to build a Garden Railroad in their back yard. It’s been an ongoing hobby project since then, and it’s been growing substantially every year:

West Station on the Point to Point Line

This is “G Scale” (around 1:22.5 scale) outdoor model railroading. It’s really a combination of three things: model railroading, building miniatures, and gardening. This recent weekend was the local club’s open-house day, and I was invited along to see many of the layouts. Each layout kind of emphasizes its own focus: some are more focused on gardening, others on the miniatures, etc.

Anyone who does anything remotely related to computers knows that every relative you have thinks you know everything there is to know about computers, and you’re destined to spend the rest of your family holidays removing spyware and running ccleaner on their computers, not to mention reassuring them that it’s OK to reboot the computer to see if the problem goes away.

Being in industrial automation, though, you never get people asking you to automate something; it’s just a little bit too abstract for most people to grasp. However, when your parents get themselves a model train set, they may not know exactly what you do for a living, but they certainly know that if you can program conveyors, robots, and cranes, you should be able to figure out how to make their trains do what they want them to do. Automatically.

Of course, as a control systems guy, you can’t look at your parents’ 24V model train set and not think about how you’d hook up a PLC to it. Especially when they offer to finance the project.

I’m happy to report that we’re progressing well. The goal is to have it running in fully automatic mode before the end of the month. I’ll post some pictures, hopefully some videos, and some technical information about how it was accomplished. Stay tuned.