Tag Archives: innovation

Automation and Software in the Next Decade

The “low intellectual property” Fashion Industry as a model for innovation?

I have to agree. Since I’ve started getting involved in open source software, I can tell you that intellectual property laws are the biggest obstacle to innovation that I face. I can say that because I took the time to consult with intellectual property lawyers on everything from trademark to copyright to open source license to software patent. Here’s what I found out: you can’t write a single line of code without violating someone’s patent, somewhere. However, the only time that person will enforce their patent claim is if you’re really successful. The only defense is to pay tens or hundreds of thousands of dollars to build up a patent portfolio of your own so you can partake in a ludicrous arms-race deterrent. On top of that, your patent portfolio offers no defense against so-called “patent trolls”: companies that only own patents and don’t write software of their own. They’re the equivalent of a terrorist cell with nuclear weapons… your nukes are ineffective because you don’t have a city to target.

Copyright is under control, mostly because it’s so much weaker than patents, and it’s free. It stops whole-sale copying of your software, but allows someone to look at your source code, learn from it, and then write it themselves without violating copyright (if they’re careful). Plus, the wide adoption of open source licenses give us lots of material that we can freely copy or integrate.

The irony is that patents were supposed to speed up innovation, but they’ve done exactly the opposite in the software industry.

Do you want to know why none of this matters though? I’m going to make a bold prognostication here:

If the last decade (have we agreed on what to call it yet? The Oh’s?) was about more crap, then the next decade will be about customized crap. Seriously, we gorged ourselves on the mountain of crappage that is sold at Wal-mart and people are starting to wake up with a hangover. We’ve already seen the desires start to change from more to better but I think we’re going to see it change to mine. People are going to want unique stuff. Things that they had a hand in customizing. Something they can use to express themselves.

This has major implications for manufacturing, so it affects the automation industry, and I think it also affects software. Just like the fashion industry, where people use clothes to express themselves, we’re about to enter an age when all the crap you sell to consumers has to have a story, a personality, and be unique. Mass production will remain for your basic staples, and China is going to continue to provide us with that, but North America and Europe are going to have to start producing bespoke crap.

What does this mean for automation? We’re going to see a rise in the build-a-bear style build-your-own-product-on-a-website and have it manufactured and sent to your door the next day. Imagine the logistical changes that need to take place to make that happen. Internet merchandising companies like cafepress already have a leg up in this department. Their merchandise is manufactured after you order it. Now imagine a product with 10,000 or a million different variations, and the automation that has to support it. Imagine the automation required to manufacture products whose specifications change as fast as fashion trends.

On the software side, we’re going to see every major application support custom 3rd party add-ins so everyone can customize their software so it works for them. I think we’re also going to see a rise in “pseudo-programming languages” that let people who aren’t programmers actually customize their applications in ways that we’ve never let them do before. Platforms that give people the power to build solutions to their own needs are going to flourish, and one-size-fits-all solutions like word processors will fade into obscurity.

So I think we’ll have two options: the vast majority of us are going to spend our time building custom things for individuals or businesses. The successful ones will produce products that people can customize themselves.

Automation Vendors on Twitter

I’ve noticed some Industrial Automation vendors on Twitter recently (not to mention some automation related magazines).

I like that trend, but here’s some commentary on who’s doing what right and wrong (from an engineer’s perspective anyway):

Tweets with useful information for customers:

Tweets aimed at shareholders, not customers:

Not taking it seriously:

An aborted attempt:

Lowering the Cost of Automation Equipment

(This is the third part of a trilogy of blog posts about what I think is the biggest roadblock preventing significant growth in the industrial automation industry. You should read Part 1: Industrial Automation Technology is Stagnant and Part 2: Why Automation Equipment Vendors Dabble in Integration first.)

I was recently making significant additions to a PLC program and the resulting program was too big to fit in the existing PLC. There weren’t many areas where the code could be made smaller without impacting readability, so I went out and priced the same series of PLC with more memory. I’m not going to name brands or distributors here (and I’d probably get in a lot of trouble if I did, because they don’t publish their prices), but I was taken aback by one thing: the only difference between these two controllers was the amount of memory (execution speed and features were the same), and it was over $1000 more. I realize this is “industrial” equipment, so the memory probably has increased temperature ratings, and it’s battery backed, but how much PLC memory can you buy for $1000? As of writing this post, the cost of 1 Gibabyte of memory for a desktop computer is less than $50.

At $50 per Gig, 20 Gigs of PLC memory? Nope.

Ok, it’s industrial, so… 1 Gig? Nope.

Try less than half a Megabyte. Yes.

Just for the sake of comparison, how much is that per Gigabyte? If half a Meg costs $1000, then the price of PLC memory is 2 Million dollars per Gigabyte. WTF? It’s not 40 times more expensive, it’s forty thousand times more expensive than commodity RAM.

Do you know why they charge two million dollars per gigabyte for PLC memory? Vendor lock-in. It’s that simple. They can charge thousands of dollars because the switching costs are insanely high. I can either buy a PLC in the same series from the same manufacturer (and they make sure there’s only one distributor in my area) with more memory, or I can replace the PLC with one from another manufacturer incurring the following costs:

  • Rewriting the program because the languages are not portable.
  • Modifying the electrical drawings significantly.
  • Rewiring the electrical panel.
  • Retraining all of my existing maintenance and engineering staff on the new platform.

So it really doesn’t matter. They could charge us $5000 for that upgrade and it’s still cheaper than the alternative. Why don’t they publish their prices? Simple, they price it by how much money they think you have, not by how much it costs them to produce the equipment. Wouldn’t we all like to work in an industry like that?

How did it get this way? In the PC industry, if Dell starts charging me huge markups on their equipment, I can just switch to another supplier of PC equipment. Even in the industrial PC world, many vendors offer practically equivalent industrial PCs. You can buy a 15″ panel mount touch screen PC with one to two gigs of RAM for about $3000. Add $500 and you can probably get an 80 Gig solid state hard drive in it. Compare that with the fact that small manufacturing plants are routinely paying $4500 for a 10″ touch screen HMI that barely has the processing power to run Windows CE, and caps your application file at a few dozen megabytes!

The difference between the PC platform and the PLC platform comes down to one thing: interoperability standards. The entire explosive growth of the PC industry was based on IBM creating an open standard with the first IBM PC. Here’s the landscape of personal computers before the IBM PC (pre 1980)*:

Here’s the market share shortly after introduction of the IBM PC (1980 to 1984):

… and here’s what eventually happened (post 1984):

Nearly every subsystem of a modern PC is based on a well defined standard. All motherboards and disk drives use the same power connectors, all expansion cards were based on ISA, or later PCI, and AGP standards. External devices used standard RS232, and later USB ports. Hardware that wasn’t interoperable just didn’t survive.

That first graph is exactly like where we are now in the automation industry with PLCs. None of the vendors have opened their architecture for cloning, so there’s no opportunity for the market to commoditize the products. Let’s look at this from the point of view of a PLC manufacturer for a moment. Opening up your platform to cloning seems like a really risky move. Wouldn’t all of your customers move to cheaper clones? Wouldn’t it drive down the price of your equipment? Yes, those things would happen.

But look at what else would happen:

  • The hardware cost of your platform would drop, so everyone would leave your competitor’s platforms and come to yours. At least you’d still be in the game, but your competitors wouldn’t even have a compatible platform anymore.
  • By dropping the price of automation equipment, the demand for complementary products would increase. IBM didn’t make its fortune directly in the PC industry, but they rode the wave and became an enterprise services company built on top of the commodity PC business.
  • The size of the industry would explode.

Why don’t we have open standards? What about IEC-61131-3? PLC vendors tell us that their controllers are IEC 61131-3 compliant, but that doesn’t mean they’re compatible. The standard only specifies what languages must be supported and what basic features those languages should have. Why didn’t the committee at least specify a common file format? You may be surprised to learn that many of the major committee members were actually from the PLC equipment manufacturers themselves. The same people who have a vested interest in maintaining their own vendor lock-in were the ones designing the standard. No wonder!

What about the new XML standard? Well, it’s being specified by the same group and if you read their documentation, you can clearly see that this XML standard is not meant for porting programs between PLC brands, but rather to integrate the PLC programming tools with 3rd party programs like debugging, visualization, and simulation tools. You can be certain that a common vendor-neutral file format will not be the result of this venture.


I see three ways this problem could be solved:

  1. An association of common manufacturers and system integrators could form and develop a truly open and interoperable standard,
  2. A PLC manufacturer could “go IBM” and open up their platform, or
  3. A single integrator or manufacturer could develop and create an open automation platform and put the rights in the public domain.

Option 1 is almost doomed to fail because it has the word committee in it, but options 2 and 3 (which are similar) have happened before in other industries and stand a good chance of success here.

In my opinion, the most likely candidate to fulfill option 2 is Beckhoff. I say this because they’ve already opened up their EtherCAT fieldbus technology as an open standard, and their entire strategy is based around leveraging existing commodity hardware. Their PLC is actually a real-time runtime on a PC based system, and their EtherCAT I/O uses commodity Ethernet chipsets. All they would really need to do is open source their TwinCAT I/O and TwinCAT PLC software. Their loss of software license revenue could be balanced by increased demand for their advanced software libraries in the short term, and increased demand for their EtherCAT industrial I/O in the longer term. Since none of the other automation vendors have a good EtherCAT offering, this could launch Beckhoff into the worldwide lead relatively quickly.

For option 3, any company that considers automation equipment to be an expense or a complementary product (i.e. manufacturing plants and system integrators) could do this. There is a long term ROI here by driving down the cost of automation equipment. Many internet companies do this on a daily basis. IBM sells servers and enterprise services, which is the natural complement of operating systems, so they invest heavily in Linux development (not to drive down the cost of Linux – it’s free – but to offer a competitive alternative to Microsoft to keep Microsoft’s software costs down). Google does everything it can to increase internet usage by the public, so it invests heavily in free services like Google Maps, Gmail, and even the Firefox web browser. The more people who surf, the more people who see Google’s advertising, and the more money they make.

If I were going to go about it, I’d build it like this:

  • Base it on commodity PC hardware (x86).
  • Pick an open source, free real time operating system like FreeRTOS.
  • Document and publish (free download) the communication protocol and file format for the automation program files (e.g. ladder logic).
  • Write a reference runtime that conforms to this communication protocol and publish it under a BSD license. It doesn’t have to be great, but it has to work and be useful.
  • Write an extensible programming environment (Windows-based) where you can develop automation programs that conform to the standard and that communicates with any runtime that conforms to the standard. Give it away free and publish it under the GNU GPL license to prevent it from being embraced and extended in a proprietary way by one vendor.

If you did that, anyone could make hardware that’s compatible with this platform, people are free to innovate on the runtime, and anyone can make improvements to the programming environment (as long as they give their changes back to the community). Instant de-facto standard.

I know some of you may be grinning right now because this is the path I’m following with my next personal software project, that I’m calling “Snap” (which stands for “Snap is Not A PLC”). However, Snap is for the home automation industry, not the industrial automation industry. If any company out there wants to take up the gauntlet and do this for industrial automation equipment, you may just change this industry and solve our biggest problem.

Discussion is welcome. Please comment below or contact me by email (e8pzrhs02@sneakemail.com).

* Graphs courtesy of ars technica.

Why Automation Equipment Vendors Dabble in Integration

(This is part 2 of a trilogy. You should read Part 1: Industrial Automation Technology is Stagnant first, and then you can read the thrilling conclusion: Lowering the Cost of Automation Equipment.)

Tell me if this sounds familiar… you’re a systems integrator and you’ve been asked to come and bid on a project. At the bid line-up meeting, you sit down in the meeting room, and you look across the table, and there’s someone from the equipment vendor XYZ’s integration services department.

Of course the customer has specified that you have to use equipment from vendor XYZ, and of course the integration services wing of vendor XYZ can purchase this equipment internally (at a discount) so they can obviously undercut your price. This always left a bad taste in my mouth, but thankfully some vendors have stopped doing this.

But ask yourself why they would do this in the first place. Integration is a cut-throat market and the margins in the equipment business are significantly higher than doing integration. What business sense does it make to expand into a low margin market and risk annoying the very companies who are out there pushing your product to the end customer?

“Commoditize Your Complements”

Everyone knows the economic principle of supply and demand. If supply goes up, price goes down, but if demand goes up, price goes up. That explains what happens when equipment vendors offer integration services and undercut the competition: the supply of integration services goes up, and prices go down. This is their goal. This bears repeating: Their goal is to reduce the price of integration services in general.

Let’s talk about competitive products vs. complementary products for a moment. When a competitor lowers their price, you have to lower yours, but when the price of a complementary product goes down, demand for your product goes up, and therefore the price of your product goes up. What are complementary products? They’re products that are often purchased together, like razors and razor blades, cars and gas, or even automation equipment and integration services.

Gillette made a fortune realizing that if they reduced the price of razors, the demand for blades would go up. When the price of gas goes down, people buy bigger cars. Likewise, the automation equipment vendors know that if they can reduce the price of integration services, the demand for automation equipment will go up, and they can charge more for their product.

While I appreciate the value of competition, it needs to be a fair market. By choking off integrators, the vendors are just promoting a race to the bottom of quality.

Personally, I don’t think it has to be this way. In future posts I’m going to discuss things that the integration industry can do to reverse this trend and stop the downward spiral.

Industrial Automation Technology is Stagnant

(This is part 1 of a trilogy. When you’re finished reading this, you can read Part 2: Why Automation Equipment Vendors Dabble in Integration and Part 3: Lowering the Cost of Automation Equipment.)

I’m not the first person to notice that industrial automation technology isn’t keeping up with the rapid pace of technological change that we’ve all grown used to. Jim Pinto commented on it years ago:

Industrial automation business seems to be stuck in the mold of being a slow-growth, stable business. This generates a mindset, perhaps even complacency that inhibits change. In my opinion it’s marketing myopia, an unwillingness to think “outside the box”.

Most major automation companies have gross-profit margins of 40-50%, the industry mindset. Many other large companies in other businesses generate much lower gross margins, in the region of 20-25%. Some of this may be accounting differences, but the broad-brush differences are there. Traditionally, automation business is based on higher gross margins and lower net-profit, and it’s difficult to think outside that box.

In the industrial market, only 2-3% is spent on R&D, with some lower than that. By contrast, many of high-tech manufacturers spend 10-15% – Cisco 15-20%, with HP, Lucent, in the 8-10% range, and Motorola and Ericsson at 10-15%. One wonders what the impact would be, if industrial automation companies doubled, or even tripled, their R&D investments.

How is it that an industry like industrial automation equipment can have margins in the 40 to 50% range when their primary customers are manufacturers, probably the most price-conscious market in the world today? I’ll tell you why: they’re smart. Here’s what they did…

Originally they all sold interchangeable parts. You built a control system out of power supplies, relays, terminal blocks and valve banks. All of these parts are available, in replacement form, from various manufacturers and distributors so competition keeps margins tight. Even if you only used P&B relays, it’s pretty easy to switch your standard to Phoenix if you can get a better deal.

Then they came out with something revolutionary: the PLC. Manufacturers jumped on it because you could take four double-door panels worth of relays and replace it with a single unit, and even reprogram it in real time! It was truly a marvel of technology, but it shifted the balance of power in the industry away from the customers and towards the vendors. Now this single piece of equipment that controls the assembly line isn’t interchangeable with other vendors. Not only that, but it requires expensive programming software, not to mention a steep ramp-up time for the maintenance and engineering departments to learn the technology. Manufacturers started standardizing on “automation platforms”. Obviously this creates vendor lock-in. Once you standardize on one technology, the vendors can pretty much count on your continued business as long as they provide good support and a clear upgrade path. The barrier to switching technologies is so great that vendors can afford to have high margins, and they don’t feel the pressure to innovate like in other markets.

Unfortunately this means manufacturers are losing. They’re not getting as much value for their money as they could be. Here are some examples of what a PLC should be able to do as we approach the year 2010 (but can’t):

  • Open an ODBC connection to a database and execute basic create, read, update and delete (CRUD) instructions, straight from ladder logic, without the need for expensive 3rd party software.
  • Send an email or text message when a machine goes down, or on any sort of event.
  • Support for Secure Socket Layer (SSL) the same technology your banking website uses. This would prevent exposure of some of our critical infrastructure to network security threats.

Another unfortunate side effect is reduced innovation in the system integration space. Let’s say you’re a system integrator and you want to develop some product for a niche market, you have to choose an automation technology to base it on. Once you’ve chosen that controls technology, say Allen-Bradley, you’re also locked in. You’re going to find it hard to sell your product to a customer that has standardized on Siemens, which means you have to invest extra effort, money, and time in re-engineering your product to work on their platform. Worse, their platform may not have all the features that your product requires, so it may not work as well. It’s not much of a product if you have to re-work the design every time, which means it’s harder to recoup your development cost.

Ultimately, automation technology is more expensive and less powerful because automation equipment vendors have stopped innovating.