Category Archives: .NET

Whole Home Energy Monitoring with the Brultech ECM-1240

This Christmas I asked Santa for a Brultech ECM-1240 whole home energy monitoring system, specifically the DUO-100 package. After reviewing various products, I really liked that this one was priced quite reasonably for the hardware, and that they published the communication protocol.

You (or a licensed electrian) install the ECM-1240 at your main electrical panel. Each ECM-1240 has 7 channels. In my case, the first channel on the first unit measures the incoming main line current. For the other 13 channels you are free to choose various circuits from your panel that you want to monitor. You can gang various circuits together into a single channel if you like (all lighting loads, for example). The device uses current transformers on each circuit for the monitoring. These are installed inside the panel. The hot wire coming out of each breaker has to go through a current transformer, so this isn’t a simple plug-in installation; there is wiring to be done.

The company distributes software for the device for free, or you can choose from various 3rd party software. Alternatively you can configure the device to send data to websites.

I’m not a fan of sending my home energy data to someone else’s server. Apart from being a huge privacy concern (it’s pretty easy to see when you’re home, and when you went to bed), I don’t want to pay a monthly fee, and I don’t want to worry about how to get my data from their server if they go out of business or decide to discontinue their product. For those reasons, I installed their software. It’s basically a flash website that I hosted on our Windows Home Server, which is conveniently the computer hooked up to the ECM-1240’s as well.

At first their software worked quite well, but over time it started to show problems. After just less than 2 months of logging, the flash program was so sluggish to load a page that it took over 2 minutes to refresh a screen. Admittedly I was running it on an older PC. Additionally, in that amount of time it had already logged about 680 MB of data. That seemed excessive. It also logged to a SQL Lite database (a single-user file-based database), and unfortunately kept the database file locked all the time. The website would end up locking the database and packets started getting buffered in the logging software until you closed down the website and released the lock.

I decided I’d just write my own software:

Power Cruncher Viewer

I’m a C# developer, so that was my language of choice. If you’re a .NET developer too, I’ll include a link to my source code at the end of this post. Note that this software isn’t commercial quality. It’s semi-configurable (channel names, and so on) but it assumes you have a 14 channel system where the first channel is the main panel input. If your system is different, you’ll have to make some modifications.

These devices I purchased use a serial port for communication, but they give you an RS232 splitter cable so you only have to use up one serial port on your PC. Their software relies on putting the device into an automatic send mode… the device itself chooses when to send packets. If the load changes significantly on one of the circuits, it triggers an immediate packet send. Unfortunately with 2 devices on the same RS232 port, you can sometimes get a collision. Their software deals with this by detecting it and ignoring the data, but I had a sneaking suspicion that once in a rare while, it ended up getting a corrupt packet, and there was at least one point where their software logged an extremely high energy reading, and it didn’t make any sense. Therefore I wrote my software to use a polling mode. It requests data from the first device, waits about 5 seconds, requests it from the other device, waits 5 seconds, and repeats. On average we get one reading about every 10 seconds from each device. Testing seemed to indicate this was about as fast as I could go because there was a reset period after you talked to one device where you had to wait for it to time out before you could address the other one. In retrospect, if you just hooked them up to 2 separate serial ports you could probably poll the data faster.

Next I had to choose how to log the data. I really wanted the data format to be compact, but I still wanted decently small time periods for each “time slice”. I also didn’t want to be locking the file constantly, so I just wanted to be able to write the data for each time slice, and be done with it. I settled on a binary format with fixed length records. Here’s how it works: each day’s data is stored in a separate file, so the data for April 1st is stored in a file called 2015-04-01.dat. The device generates values in watt-seconds. I calculated that at a maximum current of 200A (100A panel x 2 legs) x 120V, then in a 10 second time slice I should log a maximum of 240,000 watt-seconds. A 20-bit number (2.5 bytes) can store a maximum value of 1,048,575. I didn’t want to go smaller than half-byte increments. So, 14 channels at 2.5 bytes per channel gave me 35 bytes for each 10 second time slice, and since the devices generate voltage I figured I’d log the voltage as a 36’s byte just to make it an even number. Using those numbers, running 24 hours a day for 365 days a year, this will use up under 110 MB/year. Not bad. A flat binary file like this is also fast for data access. The data is small, and seeking a time slice can be done in O(1) time. After you find the first time slice you just keep reading bytes sequentially until you get to the end of the file or your last time slice. Hopefully no more 2 minute load times.

I broke up my application into multiple programs. The first is a utility program called Uploader.exe. This is a command line program that reads the data from a device and just spits it out to the screen. Useful for testing if it works.

The second is called Logger.exe and it does just that. It uses Uploader.exe to read values from the two ECM-1240 units at 5 second offsets and writes the data to XML files into a PacketLog folder. Each file has the timestamp of when it was read, and the device number to indicate which device it came from.

The third program is called PacketProcessor.exe. This program monitors the PacketLog folder and waits until it has at least one packet from each device after the end of the current time slice it’s trying to build. If it does, it calculates the values for the current time slice and appends it to the end of the current day’s data file. It then deletes any unnecessary packets from the PacketLog folder (it keeps at most one packet from each device before the end of the most recently written time slice).

To host Logger.exe and PacketProcessor.exe, I used nssm a.k.a. “the Non-Sucking Service Manager”. It’s a small command line program you can use to run another command line program as a service. Very handy.

The fourth program is called Viewer.exe. It’s the graphical interface for viewing the power data. Here’s a screenshot of data from one day (just the main panel power, click on it to make it bigger):

Main Panel Power over one day

That’s a rather typical day. The big spike around 5 pm is the oven, and you can see the typical TV-watching time is after 8 pm once the kids are in bed. On the right you can see it breaks out the power usage in kWh for each channel, and in our area we have time-of-use billing, and this actually breaks the power usage out into off-peak, mid-peak, and on-peak amounts.

The viewer program uses a line graph for all time periods under 24 hours, but switches to an hourly bar graph for periods longer than that. Here is a 48-hour time period, and I’ve changed from just the Main Panel to all the branch circuit channels instead:

Branch circuit power over 48 hours

If you ask for a time period of more than a week it switches into daily bars:

Branch circuit power by day

Pulling up two weeks of data like that was actually very fast, just a few seconds.

One final feature I added was email alerts. One of the channels monitors just my backup sump pump circuit. If it ever turns on, that’s because my main sump pump has failed, and I want to know. Through a configuration file, I configured an alarm to email me if that circuit ever records more than 5W of power in a 10 second time slice. I did a little test and it works (it does require that you have a Gmail account though).

Here’s the source code, if you’re interested:

As always, feel free to email me if you have questions or comments.

OWI-535 Robot Arm with USB Controller from C# and .NET

I got the OWI-535 “Robot Arm Edge” 5-Axis robot arm and USB Controller add-on for Christmas:



The robot arm comes as a kit that you assemble yourself, and my 3 year old and I had lots of fun putting it together (it helped to have some tiny fingers around, honestly). It comes with a manual controller that allows you to rotate all 4 joints, plus the gripper. It’s fun to play around with, but let’s be honest, everyone wants to hook it up to their computer.

Unfortunately the software that comes with the USB controller works on Windows 7, but “32-bit only”. That was a little annoying, but hey, I didn’t really want to stick with their canned software anyway. I figured I would see if I could get it to work from C#. I was surprised to find a great site by Dr. Andrew Davison in Thailand who wrote some Java code as part of his site for his book Killer Game Programming in Java (chapter NUI-6 Controlling a Robot Arm, which doesn’t appear in the book). Surprisingly he went as far as to do the inverse kinematic equations so you can give the arm a set of X,Y,Z co-ordinates (in mm) in “world frame” and it will calculate all the join angles to get to that location, and then used timed moves to get the arm into that position.

His Java library uses libusb-win32, and that library has a .NET equivalent called LibUsbDotNet. The API’s don’t appear to be the same, but thankfully I managed to find a thread on SourceForge about controlling the OWI-535 using LibUsbDotNet. So, over the course of a couple of nights, after the kids went to bed, I ported Dr. Davison’s Java code over to C# (quite easy actually) and replaced the libusb-win32 calls with LibUsbDotNet API calls. It worked!

Here is the .NET solution that I wrote called TestOwi535. I used Visual C# 2010 Express Edition to write it, so that’s free. Also, you must download and install LibUsbDotNet first and run the USB InfWizard first to generate a .inf file (you have to do this with the robot arm plugged in and turned on), and then use that .inf file to install LibUsbDotNet as the driver (technically you’re installing libusb-win32 as the driver and LibUsbDotNet is just the C# wrapper).

If you right click on the C# project in the solution explorer, you’ll find 3 options for startup object: MainClass is the original code I found in the SourceForge thread, but it’s just proof of concept code and only moves one joint in one direction for a second. The ArmCommunicator class is an interactive console application that allows you to move all joints (and control the gripper and light) by typing in keyboard commands. Finally the RobotArm class is the full inverse kinematics thing. In the last case, make sure you start with the arm at the zero position (base pointing away from the battery compartment, and all joints in-line straight up in the air, gripper pointing directly up, and gripper open). It will do a move to the table, to the front right of the arm, close the gripper, and then move back to zero and open the gripper.

Unfortunately that’s where you start to see the obvious deficiency of the arm: it has no position feedback. That means even though it tracks its position in the code, the physical position eventually drifts away from the internal position tracking, and the arm eventually doesn’t know where it is (it’s just using timed moves). One of the biggest areas where you could improve it is to change the joint rates so that it knows that the joints move faster when going down than when going up.

Still, it’s a neat little toy for the price. I’m going to start hunting around for a way to add joint position feedback, as that would really improve the performance. I also want to rewrite a new module from the ground up that allows concurrent joint moves (this one only moves one joint at a time). Ideally you want to control this thing in “gripper frame” instead of “world frame”.

Happy hacking!

Ladder Logic vs. C#

PC programming and PLC programming are radically different paradigms. I know I’ve talked about this before, but I wanted to explore something that perplexes me… why do so many PC programmers hate ladder logic when they are first introduced to it? Ladder logic programmers don’t seem to have the same reaction when they’re introduced to a language like VB or C.

I mean, PC programmers really look down their noses at ladder logic. Here’s one typical quote:

Relay Ladder Logic is a fairly primitive langauge. Its hard to be as productive. Most PLC programmers don’t use subroutines; its almost as if the PLC world is one that time and software engineering forgot. You can do well by applying simple software engineering methods as a consequence, e.g., define interfaces between blocks of code, even if abstractly.

I’m sorry, but I don’t buy that. Ladder logic and, say C#, are designed for solving problems in two very different domains. In industrial automation, we prefer logic that’s easy to troubleshoot without taking down the system.

In the world of C#, troubleshooting is usually done in an offline environment.

My opinion is that Ladder Logic looks a lot like “polling” and every PC programmer knows that polling is bad, because it’s an inefficient use of processor power. PC programmers prefer event-driven programming, which is how all modern GUI frameworks react to user-initiated input. They want to see something that says, “when input A turns on, turn on output B”. If you’re familiar with control systems, your first reaction to that statement is, “sure, but what if B depends on inputs C, D, and E as well”? You’re right – it doesn’t scale, and that’s the first mistake most people make when starting with event-driven programming: they put all their logic in the event handlers (yeah, I did that too).

Still, there are lots of situations where ladder logic is so much more concise than say, C#, at implementing the same functionality, I just don’t buy all the hate directed at ladder logic. I decided to describe it with an example. Take this relatively simple ladder logic rung:

What would it take to implement the same logic in C#? You could say all you really need to write is D = ((A && B) || D) && C; but that’s not exactly true. When you’re writing an object oriented program, you have to follow the SOLID principles. We need to separate our concerns. Any experienced C# programmer will say that we need to encapsulate this logic in a class (let’s call it “DController” – things that contain business logic in C# applications are frequently called Controller or Manager). We also have to make sure that DController only depends on abstract interfaces. In this case, the logic depends on access to three inputs and one output. I’ve gone ahead and defined those interfaces:

    public interface IDiscreteInput
    {
        bool GetValue();
        event EventHandler InputChanged;
    }

    public interface IDiscreteOutput
    {
        void SetValue(bool value);
    }

Simple enough. Our controller needs to be able to get the value of an input, and be notified when any input changes. It needs to be able to change the value of the output.

In order to follow the D in the SOLID principles, we have to inject the dependencies into the DController class, so it has to look something like this:

    internal class DController
    {
        public DController(IDiscreteInput inputA, 
            IDiscreteInput inputB, IDiscreteInput inputC, 
            IDiscreteOutput outputD)
        {
        }
    }

That’s a nice little stub of a class. Now, as an experienced C# developer, I follow test-driven development, or TDD. Before I can write any actual logic, I have to write a test that fails. I break open my unit test suite, and write my first test:

        [TestMethod]
        public void Writes_initial_state_of_false_to_outputD_when_initial_inputs_are_all_false()
        {
            var mockInput = MockRepository.GenerateStub<IDiscreteInput>();
            mockInput.Expect(i => i.GetValue()).Return(false);
            var mockOutput = MockRepository.GenerateStrictMock<IDiscreteOutput>();
            mockOutput.Expect(o => o.SetValue(false));

            var test = new DController(mockInput, mockInput, mockInput, mockOutput);

            mockOutput.VerifyAllExpectations();
        }

Ok, so what’s going on here? First, I’m using a mocking framework called Rhino Mocks to generate “stub” and “mock” objects that implement the two dependency interfaces I defined earlier. This first test just checks that the first thing my class does when it starts up is to write a value to output D (in this case, false, because all the inputs are false). When I run my test it fails, because my DController class doesn’t actually call the SetValue method on my output object. That’s easy enough to remedy:

    internal class DController
    {
        public DController(IDiscreteInput inputA, IDiscreteInput inputB, 
            IDiscreteInput inputC, IDiscreteOutput outputD)
        {
            if (outputD == null) throw new ArgumentOutOfRangeException("outputD");
            outputD.SetValue(false);
        }
    }

That’s the simplest logic I can write to make the test pass. I always set the value of the output to false when I start up. Since I’m calling a method on a dependency, I also have to include a guard clause in there to check for null, or else my tools like ReSharper might start complaining at me.

Now that my tests pass, I need to add some more tests. My second test validates when my output should turn on (only when all three inputs are on). In order to write this test, I had to write a helper class called MockDiscreteInputPatternGenerator. I won’t go into the details of that class, but I’ll just say it’s over 100 lines long, just so that I can write a reasonably fluent test:

        [TestMethod]
        public void Inputs_A_B_C_must_all_be_true_for_D_to_turn_on()
        {
            MockDiscreteInput inputA;
            MockDiscreteInput inputB;
            MockDiscreteInput inputC;
            MockDiscreteOutput outputD;

            var tester = new MockDiscreteInputPatternGenerator()
                .InitialCondition(out inputA, false)
                .InitialCondition(out inputB, false)
                .InitialCondition(out inputC, false)
                .CreateSimulatedOutput(out outputD)
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputA).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputB).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputA).TurnsOff()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputC).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputB).TurnsOff()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputA).TurnsOn()
                .AssertThat(outputD).ShouldBe(false)

                .Then(inputB).TurnsOn()
                .AssertThat(outputD).ShouldBe(true); // finally turns on

            var test = new DController(inputA, inputB, inputC, outputD);

            tester.Execute();
        }

What this does is cycle through all the combinations of inputs that don’t cause the output to turn on, and then I finally turn them all on, and verify that it did turn on in that last case.

I’ll spare you the other two tests. One check that the output initializes to on when all the inputs are on initially, and the last test checks the conditions that turn the output off (only C turning off, with A and B having no effect). In order to get all of these tests to pass, here’s my final version of the DController class:

    internal class DController
    {
        private readonly IDiscreteInput inputA;
        private readonly IDiscreteInput inputB;
        private readonly IDiscreteInput inputC;
        private readonly IDiscreteOutput outputD;

        private bool D; // holds last state of output D

        public DController(IDiscreteInput inputA, IDiscreteInput inputB, 
            IDiscreteInput inputC, IDiscreteOutput outputD)
        {
            if (inputA == null) throw new ArgumentOutOfRangeException("inputA");
            if (inputB == null) throw new ArgumentOutOfRangeException("inputB");
            if (inputC == null) throw new ArgumentOutOfRangeException("inputC");
            if (outputD == null) throw new ArgumentOutOfRangeException("outputD");

            this.inputA = inputA;
            this.inputB = inputB;
            this.inputC = inputC;
            this.outputD = outputD;

            inputA.InputChanged += new EventHandler((s, e) => setOutputDValue());
            inputB.InputChanged += new EventHandler((s, e) => setOutputDValue());
            inputC.InputChanged += new EventHandler((s, e) => setOutputDValue());

            setOutputDValue();
        }

        private void setOutputDValue()
        {
            bool A = inputA.GetValue();
            bool B = inputB.GetValue();
            bool C = inputC.GetValue();

            bool newValue = ((A && B) || D) && C;
            outputD.SetValue(newValue);
            D = newValue;
        }
    }

So if you’re just counting the DController class itself, that’s approaching 40 lines of code, and the only really important line is this:

    bool newValue = ((A && B) || D) && C;

It’s true that as you wrote more logic, you’d refactor more and more repetitive code out of the Controller classes, but ultimately most of the overhead never really goes away. The best you’re going to do is develop some kind of domain specific language which might look like this:

    var dController = new OutputControllerFor(outputD)
        .WithInputs(inputA, inputB, inputC)
        .DefinedAs((A, B, C, D) => ((A && B) || D) && C);

…or maybe…

    var dController = new OutputControllerFor(outputD)
        .WithInputs(inputA, inputB, inputC)
        .TurnsOnWhen((A, B, C) => A && B && C)
        .StaysOnWhile((A, B, C) => C);

…and how is that any better than the original ladder logic? That’s not even getting into the fact that you wouldn’t be able to use breakpoints in C# when doing online troubleshooting. This code would be a real pain to troubleshoot if the sensor connected to inputA was becoming flaky. With ladder logic, you can just glance at it and see the current values of A, B, C, and D.

Testing: the C# code is complex enough that it needs tests to prove that it works right, but the ladder logic is so simple, so declarative, that it’s obvious to any Controls Engineer or Electrician exactly what it does: turn on when A, B, and C are all on, and then stay on until C turns off. It doesn’t need a test!

Time-wise: it took me about a minute to get the ladder editor open and write that ladder logic, but about an hour to put together this C# example in Visual Studio.

So, I think when someone gets a hate on for ladder logic, it just has to be fear. Ladder logic is a great tool to have in your toolbox. Certainly don’t use ladder logic to write an ERP system, but do use it for discrete control.

Insteon and X10 Home Automation from .NET

I’ve been playing with my new Smarthome 2413U PowerLinc Modem plus some Smarthome 2456S3 ApplianceLinc modules and two old X10 modules I had sitting around.

Insteon is a vast improvement over the X10 technology for home automation. X10 always had problems with messages getting “lost”, and it was really slow, sometimes taking up to a second for the light to actually turn on. Insteon really seems to live up to its name; the signals seem to get there immediately (to a human, anyway). Insteon also offers “dual-band” technology, meaning the signals are sent both over the electrical wiring of the house, and over a wireless network. On top of this, Insteon implements “mesh networking”, acknowledgements, and retries. The mesh networking means that even if two devices can’t directly communicate, if an intermediate device can see both, it will relay the signal.

Now, where Insteon seems to have improved leaps and bounds on the hardware, the software support is abysmal. That’s not because there’s anything wrong with the API, but because they’ve put the Software Development Kit (SDK) behind a hefty license fee, not to mention a rather evil license agreement. Basically it would preclude you from using any of their examples or source code in an open source project. Plus, they only offer support if you’ve purchased their SDK.

So, I’ve decided to offer free technical support to anyone using a 2413U for non-commercial purposes. If you want help with this thing, by all means, email me, post comments at the end of this post, whatever. I’ll be glad to help you.

Let’s start by linking to all the useful information about Insteon that they haven’t completely wiped off the internet (yet):

Now how did I find all this information? Google. SmartHome (the Insteon people) don’t seem to provide links to any of this information from their home or (non-walled) support pages, but they either let Google crawl them, or other companies or organizations have posted them on their sites (I first found the modem developer’s guide on Aartech’s site, for instance). Once you get one document, they tend to make references to the titles of other documents, so you could start to Google for the other ones by title. Basically, it was a pain, but that’s how it was done.

Now, whether you buy the 2413S (serial) or 2413U (USB), they’re both using the 2412S internally, which is an RS232 device. The 2413U just includes an FTDI USB-to-Serial converter, and you can get the drivers for this for free (you want the VCP driver). It just ends up making the 2413U look like another COM port on your PC (in my case, COM4).

So, assuming you know how to open a serial port from .NET, and you got done reading all that documentation, you’d realize that if you wanted to turn on a light (say you had a switched lamp module at Insteon address “AA.BB.CC”), you’d want to send it this sequence of bytes (where 0x means hex):

  • 0x02 – start of message to PLM
  • 0x62 – send Insteon message over the network
  • 0xAA – high byte of Insteon ID
  • 0xBB – middle byte
  • 0xCC – low byte of Insteon ID
  • 0x0F – Flags (meaning: direct message, max hops)
  • 0x12 – Command byte 1 – means “turn on lighting device”
  • 0xFF – Command byte 2 – intensity level – full on

… after which the 2413U should respond with:

0x02, 0x62, 0xAA, 0xBB, 0xCC, 0x0F, 0x12, 0xFF, 0x06

… which is essentially just echoing back what it received, and adding a 0x06, which means “acknowledge”.

At that point, the 2413U has started transmitting the message over the Insteon network, so now you have to wait for the device itself to reply (if it does… someone might have unplugged it, after all). If you do get a response, it will look like this:

  • 0x02 – start of message from 2413U
  • 0x50 – means received Insteon message
  • 0xAA – high byte of peer Insteon ID
  • 0xBB – middle byte
  • 0xCC – low byte of peer Insteon ID
  • 0x?? – high byte of your 2413U Insteon ID
  • 0x?? – middle byte of your 2413U Insteon ID
  • 0x?? – low byte of your 2413U Insteon ID
  • 0x20 – Flags – means Direct Message Acknowledgement
  • 0x12 – Command 1 echo’d back
  • 0xFF – Command 2 echo’d back

If you get all that back, you have one successful transaction. Your light show now be on! Whew, that’s a lot of overhead though, and that’s just the code to turn on a light! There are dozens of other commands you can send and receive. I didn’t want to be bit-twiddling for hours on end, so I created a little helper library called FluentDwelling so now you can write code like this:

var plm = new Plm("COM4"); // manages the 2413U
DeviceBase device;
if(plm.TryConnectToDevice("AA.BB.CC", out device))
{
    // The library will return an instance of a 
    // SwitchedLightingControl because it connects 
    // to it and asks it what it is
    var light = device as SwitchedLightingControl;
    light.TurnOn();
}

I think that’s a little simpler. FluentDwelling is free to download, open-sourced under the GPLv3, and includes a full unit test suite.

It also supports the older X10 protocol, in case you have some of those lying around:

plm.Network.X10
    .House("A")
    .Unit(2)
    .Command(X10Command.On);

There are quite a few Insteon-compatible devices out there. In addition to lighting controls, there is a Sprinkler Controller, Discrete I/O Modules, a Rain Sensor, and even a Pool and Spa Controller. That’s just getting started!

Upgrading a Legacy VB6 Program to .NET

There is a lot of code out there written in VB6, running just fine. If you’re someone who has to maintain it, then at some point you’ll ask yourself, “should we just bite the bullet and upgrade this to .NET?”

There is, so far, no end-of-life issue on the horizon. VB6 applications will run on Windows 7, and Microsoft has vowed to support the VB6 runtime through the life of Windows 7. That will be a while, so there’s no hurry.

First, you need to do a cost-benefit to determine if it’s worth upgrading. That’s a pretty big task right there. What do you gain by moving to .NET? Certainly you gain a much richer ecosystem of utilities, libraries, persistence layers, test frameworks, etc. You’ll also find it easier to hire developers who have .NET on their resume. It’s pretty hard to find a copy of Visual Studio 6 these days, unless you have an MSDN subscription. .NET features like lambda expressions, LINQ, and reflection are also big productivity boosters if you spend the time to become proficient with them. These are all valid points, but they’re hard to measure.

You’re going to need to do some ballpark estimates. I’ve actually been doing some conversions lately, so I have some real experience to throw at it. Take any VB6 application, and it’ll take you 1/3 to 1/2 of the original development time to rewrite it in .NET with the same feature set (using test-driven development). That’s my estimate… do what you will with it. So, how much maintenance work are you doing, and how much more efficient would you be after the conversion?

So let’s take an application that took one programmer 6 months to write, and then you’ve been maintaining it in 50% of your time for the last year. So there are 12 months of development in the existing application. By my estimate you’ll need to spend 4 to 6 months rewriting. Let’s say you’re twice as fast after the conversion (if you didn’t have unit tests before and you use test-driven development during the conversion, the unit tests alone should make you this much more productive, not to mention the improvements in the IDE and the full object-oriented support). In that case, the payback period is 8 to 12 months of actual planned development. If you have that much work ahead of you, and you can afford to put off working on new features entirely for half that time, you’ll break even.

That’s still a really big investment. The problem is that you won’t have anything to show for it for half that time. It’s surprising how quickly management could lose faith in your endeavor if they a) don’t really understand what you’re doing and b) don’t see any tangible results for months.

There are alternatives to the all-or-nothing rewrite. First, you can use a conversion tool to convert the VB6 to VB.NET. The one that comes with Visual Studio 2005 is notoriously bad, but some of the commercially developed ones are apparently much better. Still, given VB6’s laughably bad support for the object-oriented programming paradigm, the code you get out of the conversion is going to smell more like VB6 than .NET. It will get you done faster, probably more than twice as fast, so it’s still an option. However you won’t get a chance to re-architect the software or normalize the database, etc., in the process.

The other alternative to the “big rewrite” is to do the upgrade in an “agile” manner. Take some time to break the software into smaller modules, each of which can be upgraded in about one month or less. This will significantly lengthen the amount of time it takes you to finish the project, but you’ll have something tangible to show after each month. Most managers can wait this long. This approach has its problems too: you need to write a lot of code to interact between the VB6 and .NET code. It can be tricky.

Normalizing a Database

If you’re in a position where you have a database as a backing store, and you need to make major database structure changes, this must affect your decision. The “big rewrite” is the most friendly to database changes: you just write a single conversion script that upgrades the existing database in-place, and you write your new version against the new schema. You have a clean slate, so you can clean up all the crufty problems in the old schema.

On the other hand, if you’re just using a conversion tool to automatically convert from VB6 to .NET, you can’t change the schema.

If you take the middle road (“agile”), you can change the database structure at the same time, but it’s much more difficult than in the “big rewrite”. As you upgrade each module, it makes sense to modify the database structure underlying that module, but unless you’re really lucky, you’ll have parts of other modules left in VB6-land that are dependent upon database tables that are changing. That means you’ll have the same problem anyone without a really good data access layer (or object-relational persistence layer) has when they go to change the database schema:

You have a whole bunch of code that looks like this: sql = "SELECT MY_COL1, MY_COL2 FROM MY_TABLE JOIN..."

Assuming you don’t have unit test coverage, how do you find all the places in your code that need to be changed when you normalize MY_COL2 out of one table into another? Of course you can start with a search and replace, but if you really have a database normalization problem, then you probably have duplicate column names all over the place. How many tables have a column called CODE or STATUS? There are many pathological cases where a simple text search is going to find too many matches and you’ll spend hours tracking down all the places where the code might change just because of one column being moved or renamed.

The most pathological case is where you have, for instance, two columns like CONTACT1 and CONTACT2 in the same table, and somewhere in the code it says sql = "UPDATE MY_TABLE SET CONTACT" & ContactNumber & " = '" & SomeValue & "'". You’re doing to have a hard time finding that column name, no matter what you do.

You need to develop a smarter system. I’ve tried a couple of different approaches. I tried one system where I auto-generated unique constants for all of my table and column names in my database, and then I wrote a script that went through my source code and literally replaced all of the instances of table or column names inside of strings with the constants. When I changed the database, I regenerated the list of constants, and the compiler was able to catch all the dependencies. Unfortunately, this method has some deficiencies: the resulting SQL statements are more difficult to read, and when you go and make changes to these statements you have to be disciplined enough to use the generated constants for the table and column names, or you break the system. Overall, it saves a lot of time if you have a lot of database changes to make, but costs extra time if you have to write new code.

I tried a different variation of the system where instead of replacing the table and column names in the string directly, I added auxiliary statements nearby that used the constants for the table and column names, and these would generate compile errors if a dependency changed. This made the code easier to read, but had problems of its own.

I don’t have a perfect answer for this problem, but if you have any SQL strings embedded in your legacy VB6 application, and you want to do big changes to your database, I can tell you that you must build a tool for yourself.

Summary

If you really must convert your application from VB6 to .NET then make sure you go into it with your eyes wide open. Engage management in a frank discussion. Make sure you get a strong commitment. If they waffle at all, walk away. The last thing anyone wants is a half-converted piece of software.

Still, I’m here to tell you that it is possible, and if you do your homework, there can be a real payback. Good luck!

Will TwinCAT 3 be Accepted by Automation Programmers?

Note that this is an old article and I now have more up-to-date TwinCAT 3 Reviews and now a TwinCAT 3 Tutorial.

In the world of programming there are a lot of PC programmers and comparatively few PLC programmers, but I inhabit a smaller niche. I’m a PLC and a PC programmer. This is a dangerous combination.

If you come from the world of PLC programming, like I did, then you start out writing PC programs that are pretty reliable, but they don’t scale well. I came from an electrical background and I adhered to the Big Design Up Front (BDUF) methodology. The cost of design changes late in the project are so expensive that BDUF is the economical model.

If you come from the world of PC programming, you probably eschew BDUF for the more popular “agile” and/or XP methodologies. If you follow agile principles, your goal is to get minimal working software in front of the customer as soon as possible, and as often as possible, and your keep doing this until you run out of budget. As yet there are no studies that prove Agile is more economical, but it’s generally accepted to be more sane. That’s because of the realization that the customer just doesn’t know what they want until they see what they don’t want.

It would be very difficult to apply agile principles to hardware design, and trying to apply BDUF (and the “waterfall” model) to software design caused the backlash that became Agile.

Being both a PLC and a PC programmer, I sometimes feel caught between these two worlds. People with electrical backgrounds tend to dislike the extra complexity that comes from the layers and layers of abstraction used in PC programming. Diving into a typical “line of business” application today means you’ll need to understand a dizzying array of abstract terminology like “Model”, “View”, “Domain”, “Presenter”, “Controller”, “Factory”, “Decorator”, “Strategy”, “Singleton”, “Repository”, or “Unit Of Work”. People from a PC programming background, however, tend to abhor the redundancy of PLC programs, not to mention the lack of good Separation of Concerns (and for that matter, source control, but I digress).

These two worlds exist separately, but for the same reason: programs are for communicating to other programmers as much as they’re for communicating to machines. The difference is that the reader, in the case of a PLC program, is likely to be someone with only an electrical background. Ladder diagram is the “lingua franca” of the electrical world. Both electricians and electrical engineers can understand it. This includes the guy who happens to be on the night shift at 2 am when your code stops working, and he can understand it well enough to get the machine running again, saving the company thousands of dollars per minute. On the other hand, PC programs are only ever read by other PC programmers.

I’m not really sure how unique my situation is. I’ve had two very different experiences working for two different Control System Integrators. At Patti Engineering, almost every technical employee had an electrical background but were also proficient in PLC, PC, and SQL Server database programming. On the other hand, at JMP Engineering, very few of us could do both, the rest specialized in one side or the other. In fact, I got the feeling that the pure PC programmers believed PLC programming was beneath them, and the people with the electrical backgrounds seemed to think PC programming was boring. As one of the few people who’ve tried both, I can assure you that both of these technical fields are fascinating and challenging. I also believe that innovation happens on the boundaries of well established disciplines, where two fields collide. If I’m right, then both my former employers are well positioned to cash in on the upcoming fusion of data and automation technologies.

TwinCAT 3

I’ve been watching Beckhoff for a while because they’re sitting on an interesting intersection point.

On the one side, they have a huge selection of reasonably priced I/O and drive hardware covering pretty much every fieldbus you’d ever want to connect to. All of their communication technologies are built around EtherCAT, an industrial fieldbus of their own invention that then became an open standard. EtherCAT, for those who haven’t seen it, has two amazing properties: it’s extremely fast, compared with any other fieldbus, and it’s inexpensive, both for the cabling and the chip each device needs to embed for connectivity. It’s faster, better, and cheaper. When that happens, it’s pretty clear the old technologies are going to be obsolete.

On the other side, they’re a PC-based controls company. Their PLC and motion controllers are real-time industrial controllers, but you can run them on commodity PC hardware. As long as PCs continue to become more powerful, Beckhoff’s hardware gets faster, and they get those massive performance boosts for free. Not only that, but they get all the benefits of running their PLC on the same machine as the HMI, or other PC-based services like a local database. As more and more automation cells need industrial PCs anyway, integrators who can deliver a solution that combines the various automation modules on a single industrial PC will be more competitive.

Next year Beckhoff is planning to release TwinCAT 3, a serious upgrade from their existing TwinCAT 2.11. The biggest news (next to support for multiple cores) is that the IDE (integrated development environment) is going to be built around Microsoft’s Visual Studio IDE. That’s a pretty big nod to the PC programmers… yes you can still write in all the IEC-61131-3 languages, like ladder, function block, etc., but you can also write code in C/C++ that gets compiled down and run in the real-time engine.

Though it hasn’t been hyped as much, I’m pretty excited that you can have a single project (technically it’s called a “solution”) that includes both automation programming, and programming in .NET languages like C# or VB.Net. While you can’t write real-time code in the .NET languages, you can communicate between the .NET and real-time parts of your system over the free ADS communication protocol that TwinCAT uses internally. That means your system can now take advantage of tons of functionality in the .NET framework, not to mention the huge amount of 3rd party libraries that can be pulled in. In fact, did you know that Visual Studio has a Code Generation Engine built in? It would be pretty cool to auto-generate automation code, like ladder logic, from templates. You’d get the readability of ladder logic without the tedious copy/paste/search/replace. (Plus, Visual Studio has integrated source control, but I digress…)

Will anyone take advantage?

With such a split between PC and PLC programmers, exactly who is Beckhoff targeting with TwinCAT 3? They’ve always been winners with the OEM market, where the extra learning curve can be offset by lower commodity hardware prices in the long term. I think TwinCAT 3 is going to be a huge win in the OEM market, but I really can’t say where it’s going to land as far as integrators are concerned. Similar to OEMs, I think it’s a good fit for integrators that are product focused because the potential for re-use pays for your ramp-up time quickly.

It’s definitely a good fit for my projects. I’m interested to see how it turns out.

Choose MVP over MVVM

When I first saw the Model-View-ViewModel pattern, I thought it was pretty cool. I actually wrote an entire framework and an application using the MVVM pattern. It works, and it gives you a nice separation of concerns between your Model and the rest of your application.

What’s never sat well with me is the amount of redundant and sometimes boilerplate code you have to write in your ViewModel. Assuming you have POCO objects in your domain model and that your domain model shouldn’t know anything about your ViewModel (it shouldn’t), then if you have a domain class called Customer with a Name property, chances are you’ll have a ViewModel class called Customer, with a property called Name, except that the ViewModel will implement INotifyPropertyChanged, etc. It works, but once you get down to coding, it’s a LOT of extra work.

There is an alternative out there called Model-View-Presenter. Most people claim that MVVM is a form of MVP, but once you look closely, that’s not the case (or at least, that’s now how people are using MVVM). In both MVP and MVVM architecture, the ViewModel/Presenter forms a separation between the View and Model. The difference is that in MVVM, the ViewModel works with Model objects explicitly, but in MVP both the View and the Model are abstracted services to the Presenter. Perhaps it’s clearer with an example:

class CustomerViewModel
{
    Customer wrappedCustomerModel;
    public CustomerViewModel(Customer customerToWrap)
    {
        wrappedCustomerModel = customerToWrap;
    }
    
    // Leaving out the INotifyPropertyChanged stuff
    public string Name 
    { get { return wrappedCustomerModel.Name; } }
}

class Presenter
{
    public Presenter(IView view, IModel model)
    {
        populateViewWithModelData(view, model);
        view.UserActionEvent +=
            new UserActionEventHandler((s, e) =>
            {
                model.ProcessAction(e.Action);
                populateViewWithModelData(view, model);
            });
    }
    private void populateViewWithModelData(
        IView view, IModel model)
    {
        // custom mapping logic here
    }
}

There’s at least one major benefit to the Presenter class over the ViewModel class: you can wrap the model.ProcessAction class in a try...catch block and catch all unhandled exceptions from the Model logic in one place, logging them, and notifying the user is a nice friendly way. In the ViewModel case, any property getter can throw an exception, which causes lots of problems in WPF, not the least of which is that it sometimes breaks the binding and no further updates get sent back and forth.

Now let’s look at the constructor of the Presenter again:

    public Presenter(IView view, IModel model)

Nothing says that the View the presenter is hooking into couldn’t be a ViewModel:

    public Presenter(IViewModel viewModel, IModel model)

If you do this, then the Presenter separates the ViewModel from the Model! Ok, does that sound like too much architecture? Why did we want a ViewModel in the first place? We wanted it because we wanted to make the GUI logic testable, and then use WPF’s binding mechanisms to do a really simple mapping of View (screen controls) to ViewModel (regular objects). You still get that advantage. You can create a ViewModel that implements INotifyPropertyChanged and fires off an event when one of its properties changes, but it can just be a dumb ViewModel. It becomes a “Model of the View”, which is what the ViewModel is supposed to be. Since the ViewModel then has no dependencies on the Model, you can easily instantiate mock ViewModel objects in Expression Blend and pump all the test data you want into them.

Doesn’t that mean we’ve shifted the problem from the ViewModel to the Presenter? The Presenter obviously has to know the mapping between the Model and the ViewModel. Does that mean it reads the Customer Name from the Model and writes it into the Customer Name property in the ViewModel? What have we gained?

What if the Presenter was smart? Let’s assume that IModel represents that state of some domain process the user is executing. Maybe it has a Save method, maybe an Abort method. Perhaps it has a property called CustomerAddress of type Address. Maybe it has a read-only property of type DiscountModel, an Enum. Even though we’re working against an abstract IModel interface which probably doesn’t include all of the concrete public properties and methods, we have the power of reflection to inspect the actual Model object.

What if the presenter actually generated a new AddressViewModel and populated it with data from the Model any time it saw a public property of type Address on the concrete Model object? What if it hooked up listener events, or passed in a callback to the AddressViewModel so it could be notified when the user modified the address, and it would write those changes back to the Model, then inspect the Model for changes and update the ViewModel with the results? What if when it saw an Enum property on the Model, it automatically generated a DropDownListViewModel? What if, when it sees a Save method, it generates a SaveViewModel that gets mapped to a button in the View?

Can we write a generic Presenter that can comprehend our Model and ViewModel objects? Can it even build the ViewModel for us, based on what the concrete Model object looks like, and perhaps based on some hints in a builder object that we pass in?

The answer to all these questions are “Yes.” We can use the Presenter to automate the generation of the ViewModel layer based on the look & feel of the domain model itself. I leave this as an exercise for the reader…

When not to use Agile?

I’ve had my head down working on SoapBox Snap recently (an open source, free ladder logic editor and runtime for your PC), so I decided it was a good time to come up for air and write a blog post. A lot has happened with Snap since I posted the sneak peek back in July. I’ve flushed out a good ladder logic instruction set, online debugging is working, and you can now execute the runtime as a Windows service, so it’ll keep running your logic in the background, and even auto-start when Windows starts.

It’s been a long time to get a first version out the door, but it’s always been the plan to adopt an agile workflow after release. That is, short release cycles and continuous small improvements. In fact, that’s what I’m going to talk about… why not to use agile releases during the initial development.

PLANNING: Much work remains to be done before we can announce our total failure to make any progress.

The image above is currently my September calendar picture at work, which made me think of this. Please click on the image to go to Despair Inc. and take a look at their stuff. It’s hilarious.

Writing SoapBox Snap took a lot of planning and design:

I started by tackling online programming. Downloading the entire application every time there’s a change won’t scale as the program grows. That is, how do you design a data structure such that I can modify it locally, generate a packet of data that only contains the difference between this version and that last version, transfer that change over a communication channel, and reconstruct the new version on the other end given the previous version and the difference? I created a library for building this type of data structure, and the communication protocols to make it work, and I packaged it in a library called SoapBox.Protocol.Base. Then I build a data structure for automation programs on top of that and put it in a library called SoapBox.Protocol.Automation. If you follow standard software architecture terminology, I now had my “Model”.

Then I decided to tackle how to make an extensible editor and runtime. I wanted other people to be able to extend SoapBox Snap with new ladder instructions and other features, so extensibility had to be built-in from the ground up. After looking at the various technologies, I settled on .NET’s new Managed Extensibility Framework (MEF), which was only recently released in .NET 4. After playing with it for a while, I realized that part of what I was building was applicable to anyone making an editor-like application with extension points. I decided to encapsulate the “framework” part of it into a re-usable library called SoapBox.Core, and I released it as open source and posted an article on CodeProject about how to use it. Over several months, people have started downloading, using, and even contributing changes back to SoapBox Core to improve it. We’ve setup a Q&A site for people to ask questions, get help, and give feedback.

Armed with a Model, and a Framework, I set off to build SoapBox Snap. At times I made some wrong turns, or started down some dead-ends, but I had a good vision of what I wanted to build. There were nights where I went to bed feeling like I’d banged my head against the keyboard for a few hours and accomplished nothing, but every morning brought a fresh perspective, idea, or insight that helped move the project forward. I didn’t accept any compromises on the core features that affect everything else, like undo/redo. If you don’t get undo/redo right at first, adding it in later means major architectural upheaval.

Ironically, the first problem I solved at a base level, online programming, is only half-implemented in this first version. However, since everything is already there to support full online programming, no major architectural changes have to be made to add it later. My approach was decidedly “bottom-up”. Agile is “top-down”. Would Agile have worked better for this project, or was I right to take a bottom-up approach?

I believe Agile has one failure point: interoperability (and SoapBox Snap is all about interoperability). If you have one closely-knit team doing the development and they’re the only ones that will ever interact with the edges of this application, then I think Agile works well. On the other hand, when you have extensibility points, API’s, or common file formats that 3rd parties are depending on, then doing the kind of massive refactoring that’s required to iteratively change a one-month-old barely-working application into a fully developed one is either going to break the contracts with all 3rd parties, or you’re going to have to support broken legacy interfaces for the rest of your application life cycle. Spending the extra time to build your application bottom-up, and releasing a relatively stable architecture to 3rd parties with working extensions and file formats greatly reduces the friction that Agile development would have caused.

At any rate, we’ve waited long enough. It’s almost over: look for it to be released in early October. I can’t wait to see what people will do with it. 🙂

Getting a Property Name as a String in F#

I’ve been playing around with F# recently, the functional language that shipped with Visual Studio 2010. I’m looking at using it to write an application using WPF and the Model-View-ViewModel architecture. One big requirement is for DataBinding.

When you bind the View to the ViewModel, you typically have to use the explicit name of the property on the ViewModel that you’re binding to (like “Text”). You also need the literal name of the property when you fire off (or receive) the PropertyChanged event. That’s always been a little ugly, because using the literal string means it isn’t compile-time checked. I got around it in C# using a helper class which uses reflection and lambda expressions to look at a piece of code (e.g. o => o.MyProperty) and get the name of the property as a string.

That utility class didn’t work in F#, mostly because F# lambda expressions aren’t the same base object as C# lambda expressions. I was faced with rewriting it. This is where F# seems to shine. Here’s the same “get property name” logic written in F#:

    open Microsoft.FSharp.Quotations.Patterns

    let propertyName quotation =
        match quotation with
        | PropertyGet (_,propertyInfo,_) -> propertyInfo.Name
        | _ -> ""

Here’s how you can use it:

    type myClass(p) =
        member x.MyProperty
            with get() = p

    let myObject = new myClass(1)

    let myPropertyName = propertyName <@ myObject.MyProperty @>

At the end, myPropertyName has been assigned the string value “MyProperty”. It’s a heck of a lot less code. In this case it only works if you have an existing object to run it against. However, you can modify the propertyName function to make it recursively dig through the Lambda and find the PropertyGet:

    let rec propertyName quotation =
        match quotation with
        | PropertyGet (_,propertyInfo,_) -> propertyInfo.Name
        | Lambda (_,expr) -> propertyName expr
        | _ -> ""

    let myPropertyName = propertyName <@ fun (x : myClass) -> x.MyProperty @>

Now you don’t need to have an instance of the class lying around to get the property name.

Phidgets!

I got ahold of some Phidgets on the weekend. These are basically USB I/O devices for amateur robotics, but I’m looking at their use in the home automation space.

Phidgets Interface Kit 8/8/8
They have drivers for lots of operating systems, and APIs for almost any programming platform under the sun.

Getting the I/O connected and controlled from a .NET application was a breeze, including the Interface Kit 8/8/8 and the little R/C Servo Controller. That little servo could certainly move a damper in a heating duct, and they have lots of environmental sensors. The wheels have started turning…