Earlier this year I reviewed TwinCAT 3 and I admit that it was a less than stellar review. Up to now TwinCAT 3 has seemed like a beta all the way.

Well, I’m pleased to say that after using and deploying TwinCAT 3.1 for several weeks, it’s a significant improvement over its predecessor, and brings it into the realm of “release-quality” software. There are several improvements I’ll try to highlight.

64-bit! Finally! Yes, it’s so nice to get rid of the old 32-bit copy of Windows 7 and get back to the world of 64-bit computing.

The TwinSAFE safety editor is so much better. While the old one would take several minutes to verify my safety program before downloading it, and several minutes to even get online with the safety controller, this new version does it in seconds. It was so fast in comparison that I thought it didn’t work the first time. Thank goodness because that was a major shortcoming of the old version.

I’m pleased to report that Beckhoff has made changes to the source code file format to make them more friendly to source control applications (like Subversion or Mercurial). Doing a “diff” between 2 file versions now gives you a really good idea of what changed, rather than some obscure XML code. This is one of the first things I complained about and I didn’t expect it to get changed, so I’m really happy to see that. (Also remember that TwinCAT 2 files were completely unfriendly to Source Control; they were just a big binary blob!) Note that when you upgrade your TwinCAT 3.0 solution to TwinCAT 3.1, it has to do a conversion, and it’s one-way. Keep a backup of your old version just in case. In our case, the upgrade went well, except for references, but that was ok…

The old library manager in each PLC project has been replaced by a References folder, which you will be familiar with if you do any sort of .NET programming. You can now add and remove library references right in the solution tree, which is nice. That’s one thing that didn’t get upgraded automatically. I had to remove my old references to the TwinCAT 3.0 standard libraries and re-add them as TwinCAT 3.1 libraries. Not a big deal once I figured out what was causing the build error.

The scope viewer has been removed from the system tray icon menu and pulled into the Visual Studio interface. I think this is just part of their attempt to pull everything into one environment, which I applaud.

Unfortunately I still had one blue-screen-of-death when I tried to do an online change, but I couldn’t reproduce that problem. I’m using a commercial desktop computer, not an industrial PC for my testing, and Beckhoff says they won’t support the software if it isn’t installed on Beckhoff hardware. Let that be a warning to you: even though you pay significantly more for the software license in order to run it on non-Beckhoff hardware, they will not warranty software bugs in that case. That doesn’t mean you can’t get local support, but it does mean that if you have a legitimate bug report, they won’t help you unless it can be reproduced on Beckhoff hardware.

While upgrading, I also upgraded the operating system on the PC from Windows 7 32-bit to Windows 7 64-bit. That created a small problem because it caused the Beckhoff generated system ID to change, and the license is attached to the system ID. That means I have to go through generating a license file and applying the response file again. Hopefully there are no problems with duplicate licensing, as it is the same physical computer. I’m still waiting for a response though.

I did run into a couple gotchas during the upgrade. First, TwinCAT 3.1, for whatever reason, requires Intel’s Virtualization Technology Extentions (VTx) to be enabled in the BIOS, and mine weren’t. It turns out that I needed to flash the BIOS to even get that option. Secondly, we use Kaspersky Endpoint Security version 8 for our enterprise anti-virus solution. It turns out that both version 8 and version 6 prevent TwinCAT 3.1 from booting the PLC runtime into Run Mode on startup. Without Kaspersky installed it worked fine. I eventually tried Microsoft Security Essentials (the free Anti-virus solution from Microsoft) and that seems to work well. Some anti-virus is better than none, I figure.

The upgraded system has been in production for nearly 2 weeks, and seems to be working well (no problems that I can attribute directly to TwinCAT, at any rate).

In summary, if you’re looking to jump into TwinCAT 3, I know I said to wait in my last review, but I now think that TwinCAT 3.1 is a good solid base if you’re looking to get your feet wet.

review · twincat

As a licensed P.Eng., I recently received a copy of a draft version of a document regarding engineering practices for the design of safety critical software. It seemed to be well thought out and written, but one of the last statements caught me by surprise. It basically said that your design should prevent Stuxnet-like attacks.

I think it’s great that the profession is starting to take this stuff seriously, but I was a bit shocked at the implication of this document. Primarily I was concerned because Stuxnet was nothing like an ordinary attack. Defending against it would be like defending your office building against a laser guided bomb. Stuxnet was a nation-state-backed attack with huge funding and resources, going after a military/industrial target.

What does that mean for us practicing engineering in, say, the automotive industry? Do we just get to brush the whole cybersecurity thing aside because a Stuxnet-like attack against our facility has absolutely zero chance of ever being launched? It doesn’t seem like that’s what the document was trying to say.

In fact, is Stuxnet even a relevant example for “Safety Critical Software”? Did Stuxnet actually affect any safety critical systems, like a Safety PLC? All it did was damage equipment. Was anyone hurt? Was anyone in danger of being hurt? I guess I just disagree with the use of Stuxnet as an example here.

That’s not to say cybersecurity doesn’t need some attention from our profession. In particular, consider the case of so-called “Safety PLCs”. Here you have a device, typically sitting on some kind of network, definitely responsible for safety critical systems. There are at least 2 pieces of software installed on a Safety PLC: the firmware (the operating system, communication system, and runtime of the device) and the user-configurable logic that you download when you’re programming the system. Compromise of either of these pieces of software certainly poses a grave risk to human safety.

The user-configurable logic is supposed to be password protected. This is one step up from normal PLCs which rarely offer any kind of authentication. Now, when TUV, or whoever, is certifying these devices, they are definitely checking that once it’s locked, you can’t update the user-configurable software without a password. In fact, TUV has access to the design, so I *hope* someone there is well versed in cyber security and knows what they’re talking about. For instance, if it was found that authentication was being done in the programming software (client application) and not on the device itself, it would be completely useless security, because someone could easily bypass it. But let’s assume that nobody building these things is doing anything that blatantly wrong.

So even if the client/device protocol is rock solid, is the firmware bulletproof? I think this is unlikely. Every time a security researcher runs even the most basic of security audits against PLCs they usually find tons of obvious exploits. You have to realize that most of these devices are just embedded computers running off-the-shelf real-time operating systems like VxWorks. Usually it’s using the standard VxWorks networking protocols, which are sometimes known to have vulnerabilities (in certain versions) and are unfortunately rarely updated in the field. That’s not even counting deliberate backdoors and debug code left in the software. Does TUV do a security audit of every design? Maybe, but I doubt it. Even if they do, are they doing repeated security audits against older devices? When new vulnerabilities are found in these embedded operating systems that impact existing devices, are they requiring them to be recalled? I’ve never heard of such things.

I’m pretty sure the only reason we haven’t heard about this yet is because even security researchers who’ve woken up to the concept of PLCs and SCADA systems in the past 2 years still have no idea that Safety PLCs actually exist. I don’t think it’ll be too long until they find out.

Designing a device like this that’s intrinsically secure against hacking seems almost impossible to me. Whatever software module is responsible for receiving the user-configurable safety program and storing it in the persistent memory of the device is necessarily network-facing. Any vulnerability in that communication module could be exploited, and then you have the ability to bypass all the security protocols on the device, and write whatever user-configurable safety program you want.

So if this document says we have to design our Safety Critical systems to withstand a Stuxnet-like attack, and all the Safety PLCs we could use are likely vulnerable via communication module software exploits, and even firewalls and airgaps don’t seem to be able to defend against this kind of attack (just ask the employees of the Natanz facility in Iran), then where does that leave us? Is a P.Eng. supposed to just throw up their hands and say “sorry, it can’t be done?” Not likely.

safety · security

So back in 2010 I wrote about my first impression of TwinCAT 2 and later that same year I wondered if automation programmers would accept TwinCAT 3. I was lucky enough to be involved in the TwinCAT 3 beta, and now that the 32-bit version of TwinCAT 3 is available for general release we’ve deployed 2 production systems based on TwinCAT 3, and will likely deploy more in the future. What follows are my impressions of the current state of TwinCAT 3 based on our experiences with those 2 systems.

I think the best way I can describe TwinCAT 3 to the non-initiated is by comparing it with Allen-Bradley’s ControlLogix platform with their RSLogix 5000 programming environment. I say that because I’m familiar with that platform, and so are most of my North American readers (I assume). It speaks well of Allen-Bradley that they are the de-facto default control system platform around here.

We often fall back on car analogies, and I don’t want to break with tradition. If ControlLogix is the Ford Taurus of control systems (common, reliable, with lots of performance for most tasks, lots of room, and fairly maintenance free) then TwinCAT 3 is something like the Rally Fighter. That is, it’s road legal, fairly rare, requires lots of TLC and understanding and may not be as reliable, but will take you places you’ll never get to go in a Ford Taurus.

When it comes to speed, TwinCAT 3 with Beckhoff’s EtherCAT I/O is a beast. I can’t stress this enough. We’re running both production systems of TwinCAT 3 with both a 0.5 millisecond logic scan time and a 0.5 millisecond I/O bus scan time and we’re only using about 10% to 15% of the available horsepower of each system. You can see and react to things in the TwinCAT 3 system that you’ll just miss in a ControlLogix processor. For instance in one case we’re driving an output off of an absolute encoder and the repeatability of turning on that output is much better than anything we’ve seen with any other controller.

Furthermore, data accessibility is light years beyond any traditional PLC. In one test I moved a 400 kB block of data from the real-time (ladder logic program) to a .NET program running under windows on the same PC and all I can say is that it’s nearly instantaneous. That’s an advantage of having the HMI and real-time executing on the same physical hardware.

That’s not even getting into the new C++ integration (which I haven’t used).

Any TwinCAT 3 vs. ControlLogix system comparison will also certainly favor the Beckhoff solution when it comes to price. At least I can’t seen any case yet where that’s not true by a significant margin. I can’t say exact prices, obviously, but I’m confident that’s generally a true statement.

Does that mean I think the competition is a hands-down blow-out in favor of TwinCAT 3? No. In fact if you’re considering trying TwinCAT 3 I can’t even go so far as to give you my blessing right now. It has problems.

TwinCAT 3 crashes with a blue screen. Regularly. There, I said it. That’s the dirty secret. Everyone’s fears about PC-based control on the factory floor were around stability and in our experience TwinCAT 3 isn’t stable yet. This is odd to me because we have another system with TwinCAT 2 and it’s solid as a rock. Unfortunately our TwinCAT 3 system crashes with a blue screen regularly, and the crash report always shows that it’s some kind of memory violation in their Tc*.sys files, which are the system files responsible for running their real-time system under ring 0 of the OS (as far as I understand). I’ve never even seen blue screens with Windows 7 until trying out TwinCAT 3. There are actually two different times that it crashes: (1) randomly, and (2) when I try to do an online change.

Beckhoff’s response was that we were running it on 3rd party hardware. They loaned us a Beckhoff industrial PC to try. We tried it for a week and we didn’t see any random crashes, but it still crashed the real-time when I tried to do an online change, and it also crashed the IDE almost every time I recompiled the program. In fact that’s the reason we had to stop the test with Beckhoff’s hardware after only one week. I needed to compile a change and couldn’t get it to compile. It did work on our 3rd party PC (an HP desktop PC).

Now, I don’t think these are insurmountable problems. Beckhoff is still coming out with new versions on a regular basis. Their new support for 64-bit windows operating systems is on the horizon. TwinCAT 2 seems stable and I’m sure TwinCAT 3 will get there eventually. However, for the moment, if you’re considering the plunge, I suggest waiting about a year before bothering to check it out. If you just need the speed, consider TwinCAT 2, as even though it doesn’t take advantage of multiple cores it will likely do what you need and is a much more mature product at this point.

plc · review · twincat

Are you old enough to remember the early 90′s? Almost nobody was “online” then. It’s hard to believe how much the internet has changed modern life since then. I just went through my bookshelf and got rid of all the reference books (which I could have done years ago) because the internet makes them obsolete.

Do you remember having to look up some arcane technical detail in a book? Do you remember how painful that was? Do you remember connecting to a BBS using a phone line and a 2400 baud modem? I do.

Back then when we tried to imagine the future, we (as geeks) really got it wrong, and surprisingly so. We were the ones who were supposed to see the possibilities, yet we consistently imagined the things we wanted to do with the technology instead of the things the average person wants to do with it. You see, geeks aren’t average people, so we’re sometimes blind to the obvious.

As geeks, we imagined having our own robot to do our bidding, or some kind of virtual reality headset for driving cars around on Mars. In reality, the killer app for the masses was the ability to take a picture of something, write a short note about it, and send it to all your “friends”. The funny thing is, if you asked a geek in the early 90′s if that feature could be the basis of a company valued in the billions of dollars, they’d have laughed you out of the room. The idea just wasn’t that revolutionary. After all, we’d had fax machines for several years, and email/usenet were already out there. Geeks already had access to this technology, and thought nothing of it.

Average people, if they even saw this technology, didn’t see what was under their noses either, simply because it wasn’t sexy yet. Let me be clear, when I say “sexy” I mean it literally. The average young person isn’t like a geek. They don’t have robots and virtual reality sets on their mind, they want to know who’s going to be at the party on Saturday, who said what about who, and if so and so is available. Modern social media succeeds because it helps people satisfy the most basic of “normal” wants.

Where am I going with this? When you look at the bleeding edge of technological progress today it’s easy to spot the trends. Open source hardware has really taken off, specifically 3D printing and other “home” fabrication technologies. We’re also seeing an explosion of sensors, not just in your phone, but also plugged into consumer devices, with most of them uploading their data to central servers (e.g. the popular Nest Home Thermostat).

We all want to know what the next big thing will be, and we’re hopelessly bad at predicting it, but we still like to play the game. If we want to know where this new hardware renaissance is leading, we need to look at the space where these new technologies intersect the wants of average people. Only about 1% of users are the kind that participate by actually creating new content, so I definitely don’t see a future where everyone is designing new things in Google Sketchup and printing them out on their 3D printer they bought at Wal-mart. The thing about 3D printers is that they print things we can buy much more cheaply (little plastic trinkets), especially if it’s the type of thing that a lot of people want, so there’s an economy of scale to support manufacturing it.

There is something that everybody wants to create at home, and can’t be mass produced in a factory on the other side of the world: freshly prepared food. Gourmet food, even. People pay big bucks for fancy coffee makers that make fancy coffee from packets with barcodes containing individualized brewing instructions. It’s not a big stretch to imagine a machine that can whip up fancy cocktails, or dozens of fancy hors d’oeuvres for a party, or even print out your kids’ drawings on their Christmas cookies with icing from a 3D printer. Maybe a gizmo that carves an apple into the shape of their favorite cartoon character? (Of course, you know these will be re-purposed to make snacks for bachelorette parties, right?)

That’s the easy stuff. Honestly, Eggs Benedict seems like it’s well within the range of possibilities, and who wouldn’t want to wake up on Sunday to the sizzle of eggs and bacon, along with the usual coffee brewing?

Of course, if we automate the food preparation and we’re collecting all this data with sensors, it won’t be long until we’re automating the control of our caloric intake.

That’s my guess anyway: all this new technology’s going to end up in your kitchen. Which is cool with me (that doesn’t mean I can’t still have my 3D printer…)

3d-printer · food · future

I happen to have been in the market for some switching 24VDC power supplies that could source some decent current and put up with a little abuse. It turns out there’s no need to spend hundreds of dollars for one of the fancy DIN-rail mounted models when all you need is a box to sit on your test bench.

You can order them directly from Chinese suppliers, but it’s more convenient to order from someone domestic. I’ve found a good place to look is Amazon (a close second is E-Bay, but you’re really buying from China in about 50% of cases there). On Amazon you can get a 15A 24VDC power supply for well under $30:

Obviously this is a “buyer beware” type of situation, but I’ve purchased 3 similar power supplies so far, and haven’t had any problems. In one case the equipment under test shorted out due to a blown capacitor, and the power supply turned itself off until I cycled the power to the supply. It’s still working fine.

You may think that a $300 brand name power supply is what you need, but if you don’t need five 9′s of reliability then I’d like to point out you can buy ten of these cheap supplies for the same price, so buy 3 and keep 2 on the shelf, and spend the other $210 on other cool gadgets!

hardware · power-supply

Do you know how ERP software is sold? It’s fairly straightforward. Every CEO has a similar problem… they want to know how much everything in their company costs, vs. how much value it brings to the company. ERP vendors meet with executives and show them pretty graphs.

Then, of course, the company signs a contract with the ERP vendor for hundreds of thousands of dollars to install, configure, and support the system for one year. By the time the project is complete it costs twice what it was supposed to and all the end users are frustrated because it’s so tedious to use.

How does this happen? Why did the CEO think they were buying into a low-risk off-the-shelf product?

The CEO was lulled into a false sense of security by the pretty graphs. Any programmer can tell you that charts are a solved problem. If you give any programmer a database full of rich and meaningful data, they can whip up pretty reports, even very complicated ones, in a matter of days. The hard part is filling the database with accurate and complete information! If you buy an off-the-shelf ERP application, getting the data into their system is the hard part. You either need to write tons of custom code to copy the data from existing systems, or you need to change your business workflows to conform to this new software.

Consider another favorite of enterprise software applications: OEE. In order to calculate your true OEE you need the following:

1. Availability
2. Performance
3. Quality

It’s that first item, Availability, that’s really interesting. Basically it’s the ratio of actual equipment uptime vs. scheduled production time. Let’s say you’re buying an “automated OEE solution,” so it’s going to pull the status of the equipment (running or not running) from the PLC. It doesn’t really matter because that isn’t enough information to calculate your Availability. You need a production schedule to begin with. Who enters this data? Does it already exist? In what form? Excel? A proprietary ERP or MES system? What custom code has to be written to get this production schedule into a format that the “shrink-wrapped” OEE software can understand? Does the company have to convert to using the OEE software’s proprietary scheduling system? Do we have to enter the schedule twice?

If you have to do all that work, why are you buying an OEE software package? If the production schedule is already in your ERP system, and you have in-house expertise to get the running status out of the equipment, why bother converting the schedule into some other format? The equations for OEE are grade 5 math. Any programmer can make pretty OEE graphs if they already have the data they need. What value does the OEE software actually bring?

When you think about enterprise software, remember that most of the real value is in the accuracy and meaningfulness of the data in the database. Money should be invested in acquiring, storing, curating, and documenting that data. Once you’ve accomplished that, then you can add your whiz-bang charts. Don’t go putting the cart before the horse.

No tags

If you’ve ever had to do any VB6 programming, you know one (of many) shortcomings is VB6′s pitiful error handling support. The problem is that you want to know what line number an error occurred on, but the ERL function only gives you that information if you have line numbers on your entire project, and nobody wants to work with a project full of line numbers.

There are tools like MZ-Tools that allow you to add and remove line numbers with a couple of mouse clicks, but that’s not ideal. What you really want is the ability to add line numbers as an automated step during the build process. Thankfully someone went and created such a tool (written in VB6 with the source code provided). Unfortunately I’ve lost the original location that I downloaded it from, and the original author. Until I can find that page, here is a zipped up copy of that tool:

Tools_ZIP Code_csLineNumber

It provides both a GUI and command line interface. It also has instructions on how to integrate it into the Windows shell so you can just right click on the .vbp file and run the tool. Note that on Windows 7 you likely have to run it as an Administrator.

vb6

Back in November I published a blog post about Finding Internet-Connected Industrial Automation Devices and one of the scariest things I found was a wind turbine in Oklahoma with no apparent authentication.

Recently Dan Tentler took this several steps further and posted his video from the LayerOne 2012 security conference, where he shows you a vast array of non-secure devices connected to the internet, much of which can interact physically with the real world, including control systems. Here’s his extremely fascinating video, and it’s worth watching all 45 minutes (note that he also has a screenshot of the Endurance Wind Turbine interface that I found in my original post):

security

There’s a recurring theme on this blog when I talk about programming. It’s a skill you don’t learn in school, but which separates recent graduates from employees with 5 years of experience: how to manage complexity.

The reason you never learned how to manage complexity in nearly 20 years of school is that you never actually encountered anything of sufficient complexity that it needed managing. That’s because at the beginning of each term/semester/year you start with a blank slate. If you only ever spend up to 500 hours on a school project, you’ll never understand what it feels like to look at a system with 10,000 engineering hours logged to it.

That’s when the rules of the game change. You’ve only ever had to deal with projects where you could hold all the parts and variables in your head at one time and make sense of it. Once a project grows beyond the capability of one person to simultaneously understand every part of it, everything starts to unravel. When you change A you’ll forget that you had to change B too. Suddenly your productivity and quality start to take a nosedive. You’ve reached a tipping point, and there’s no going back.

It isn’t until you reach that point that you understand why interfaces are useful and why global variables are bad (not to mention their object-oriented cousins, the singleton).

The funny thing is that you learn about all the tools for managing complexity in school. When I was in high school I was taught about structured programming and introduced to modular programming. In university my first programming class used C++ and went through all the features like classes, inheritance and polymorphism. However, since we’d never seen a system big enough to need these tools, most of it fell on deaf ears. Now that I think about it, I wonder if our professors had ever worked on a system big enough to need those tools.

The rules of managing complexity are:

• Remove unnecessary dependencies between parts of your system. Do this by designing each module against an abstract interface that only includes the functionality it needs, nothing else.
• Make necessary dependencies explicit. If one module depends on another, state it explicitly, preferably in a way that can be verified for correctness by an automated checker (compiler, etc.). In object-oriented programming this is typically done with constructor injection.
• Be consistent. Humans are pattern-recognition machines. Guide your reader’s expectations by doing similar things the same way everywhere. (Five-rung logic, anyone?)
• Automate! Don’t make a person remember any more than they have to. Are there 3 steps to make some change? Can we make it 2, or better yet, 1?

It would be interesting if, in university, your second year project built on your first year project, and your third year project built on your second year project, etc. Maybe by your fourth year you’d have a better appreciation for managing complexity.

complexity · quality

Millions of people play the lottery every week. Some just buy a “quick pick” ticket (which lets the computer pick for them). Some faithfully play the same numbers over and over. Some have clever schemes.

Enough people play that even though the odds are long, a few people win. Invariably the winners are interviewed, and people are interested in what made them special. Did they have a system? Did they have a “feeling” that they would win today?

It makes a good story, and we all love a good story. We’re wired for it.

If a lottery winner wrote a book called “How I won the lottery, and you can too!” would you read it? (Believe it or not, many people make a living selling such advice — it’s usually some kind of psychic deal.) Of course you wouldn’t; you’re not foolish.

But you have to wonder, aren’t most of the books in the success section of the bookstore just by people who lucked out once? Microsoft and Google both only had one major breakthrough (an insanely great licensing deal for DOS, and a great search engine respectively). Everything else they did was through incremental building and improvement. Yet everyone seems to be focused on the big win. How can I make my numbers get drawn this week in the lottery? What’s the secret?

There is no secret, of course. Roll the dice. Be prepared to lose.

The bulk of technological progress, and even financial progress, is incremental. Just look at the difference between a good investment and a bad investment. A company with a 5% rate of growth feels slow and plodding. If a medium sized company can post a 15% growth rate every year, it’s moving so fast it probably feels like it’s running off the rails.

Even so, the company growing at 5% will still double in 14 years (handy trick: take 70 and divide by the percent growth to get the approximate doubling time). Computer power doubles about every 18 months, but the industry was never driven by people looking to win a lottery. They just kept making die sizes smaller, overcoming each technical barrier as it arose, and packing more transistors on a chip. DNA sequencing technology is growing at an even faster rate.

Keep asking yourself “How can I cut this machine’s cycle time in half? How can I cut the scrap by a third? What similar product could we produce that has a higher demand? How can I cut energy use by 10%?”

If you do this enough, you’ll still win, and the odds are a lot better.

No tags