There’s a tool in the PC programming world that nobody would live without, but almost nobody in the PLC or Automation World uses: version control systems. Even most PC programmers shun version control at first, until someone demonstrates it for them, and then they’re hooked. Version control is great for team collaboration, but most individual programmers working on a project use it just for themselves. It’s that good. I decided since most of you are in the automation industry, I’d give you a brief introduction to my favourite version control system: Mercurial.

It’s called “Mercurial” and always pronounced that way, but it’s frequently referred to by the acronym “Hg”, referring to the element “Mercury” in the periodic table.

Mercurial has a lot of advantages:

• It’s free. So are all of the best version control systems actually. Did I mention it’s the favourite tool of PC programmers? Did I mention PC programmers don’t like to pay for stuff? They write their own tools and release them online as open source.
• It’s distributed. There are currently two “distributed version control systems”, or DVCS, vying for supremacy: Hg and Git. Distributed means you don’t need a connection to the server all the time, so you can work offline. This is great if you work from a laptop with limited connectivity, which is why I think it’s perfect for automation professionals. Both Hg and Git are great. The best comparison I’ve heard is that Hg is like James Bond and Git is like MacGyver. I’ll let you interpret that…
• It has great tools. By default it runs from the command line, but you never have to do that. I’ll show you TortoiseHg, a popular Windows shell that lets you manage your versioned files inside of a normal Windows Explorer window. Hg also sports integration into popular IDEs.

I’ll assume you’ve downloaded TortoiseHg and installed it. It’s small and also free. In fact, it comes bundled with Mercurial, so you only have to install one program.

Once that’s done, follow me…

First, I created a new Folder on my desktop called My New Repository:

Now, right click on the folder. You’ll notice that TortoiseHg has added a new submenu to your context menu:

In the new TortoiseHg submenu, click on “Create Repository Here”. What’s a Repository? It’s just Hg nomenclature for a folder on your computer (or on a network drive) that’s version controlled. When you try to create a new repository, it asks you this question:

Just click the Create button. That does two things. It creates a sub-folder in the folder you right-clicked on called “.hg”. It also creates a “.hgignore” file. Ignore the .hgignore file for now. You can use it to specify certain patterns of files that you don’t want to track. You don’t need it to start with.

The .hg folder is where Mercurial stores all of its version history (including the version history of all files in this repository, including all files in all subdirectories). This is a particularly nice feature about Mercurial… if you want to “un-version” a directory, you just delete the .hg folder. If you want to make a copy of a repository, you just copy the entire repository folder, including the .hg subdirectory. That means you’ll get the entire history of all the files. You can zip it up and send it to a friend.

Now here’s the cool part… after you send it to your friend, he can make a change to the files while you make changes to your copy, you can both commit your changes to your local copies, and you can merge the changes together later. (The merge is very smart, and actually does a line-by-line merge in the case of text files, CSV files, etc., which works really well for PC programming. Unfortunately if your files use a proprietary binary format, like Excel or a PLC program, Mercurial can’t merge them, but will at least track versions for you. If the vendor provides a proprietary merge tool, you can configure Mercurial to open that tool to merge the two files.)

Let’s try an example. I want to start designing a new automation cell, so I’ll just sketch out some rough ideas in Notepad:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder

I save it as a file called “Line 1.txt” in my repository folder. At this point I’ve made changes to the files in my repository (by adding a file) but I haven’t “committed” those changes. A commit is like a light-weight restore point. You can always roll back to any commit point in the entire history of the repository, or roll forward to any commit point. (You can also “back out” a single commit even if it was several changes ago, which is a very cool feature.) It’s a good idea to commit often.

To commit using TortoiseHg, just right click anywhere in your repository window and click the “Hg Commit…” menu item in the context menu. You’ll see a screen like this:

It looks a bit weird, but it’s showing you all the changes you’ve made since your “starting point”. Since this is a brand new repository, your starting point is just an empty directory. Once you complete this commit, you’ll have created a new starting point, and it will track all changes after the commit. However, you can move your starting point back and forth to any commit point by using the Update command (which I’ll show you later).

The two files in the Commit dialog box show up with question marks beside them. That means it knows that it hasn’t seen these files before, but it’s not sure if you want to include them in the repository. In this case you want to include both (notice that the .hgignore file is also a versioned file in the repository… that’s just how it works). Right click on each one and click the Add item from the context menu. You’ll notice that it changes the question mark to an “A”. That means the file is being added to the repository during this commit.

In the box at the top, you have to enter some description of the change you’re making. In this case, I’ll say “Adding initial Line 1 layout”. Now click the Commit button in the upper left corner. That’s it, the file is now committed in the repository. Close the commit window.

Now go back to your repository window in Windows Explorer. You’ll notice that they now have green checkmark icons next to them (if you’re using Vista or Windows 7, sometimes you have to go into or out of the directory, come back in, and press F5 to see it update):

The green checkmark means the file is exactly the same as your starting point. Now let’s try editing it. I’ll open it and add a Zone 4 to the conveyor:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
    - Zone 4
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder

The icon in Windows Explorer for my “Line 1.txt” file immediately changed from a green checkmark to a red exclamation point. This means it’s a tracked file and that file no longer matches the starting point:

Notice that it’s actually comparing the contents of the file, because if you go back into the file and remove the line for Zone 4, it will eventually change the icon back to a green checkmark!

Now that we’ve made a change, let’s commit that. Right click anywhere in the repository window again and click “Hg Commit…”:

Now it’s showing us that “Line 1.txt” has been Modified (see the M beside it) and it even shows us a snapshot of the change. The box in the bottom right corner shows us that we added a line for Zone 4, and even shows us a few lines before and after so we can see where we added it. This is enough information for Mercurial to track this change even if we applied this change in a different order than other subsequent changes. Let’s finish this commit, and then I’ll give you an example. Just enter a description of the change (“Added Zone 4 to Conveyor”) and commit it, then close the commit window.

Now right-click in the repository window and click on Hg Repository Explorer in the context menu:

This is showing us the entire history of this repository. I’ve highlighted the last commit, so it’s showing a list of files that were affected by that commit, and since I selected the file on the left, it shows the summary of the changes for that file on the right.

Now for some magic. We can go back to before the last commit. You do this by right-clicking on the bottom revision (that says “Adding initial Line 1 layout”) and selecting “Update…” from the context menu. You’ll get a confirmation popup, so just click the Update button on that. Now the bottom revision line is highlighted in the repository explorer meaning the old version has become your “starting point”. Now go and open the “Line 1.txt” file. You’ll notice that Zone 4 has been removed from the Conveyor (don’t worry if the icons don’t keep up on Vista or Win7, everything is working fine behind the scenes).

Let’s assume that after the first commit, I gave a copy of the repository to someone, (so they have a copy without Zone 4), and they made a change to the same file. Maybe they added some detail to the Nut Feeder section:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder
    - 120VAC, 6A

Then they committed their change. Now, how do their changes make it back into your repository? That’s by using a feature called Synchronize. It’s pretty simple if you have a copy of both on your computer, or if each of you have a copy, and you also have a “master” copy of the repository on the server, and you can each see the copy on the server. What happens is they “Push” their changes to the server copy, and then you “Pull” their change over to your copy. (Too much detail for this blog post, so I’ll leave that to you as an easy homework assignment). What you’ll end up with, when you look at the repository explorer, is something like this:

You can clearly see that we have a branch. We both made our changes to the initial commit, so now we’ve forked it. This is OK. We just have to do a merge. In a distributed version control system, merges are normal and fast. (They’re fast because it does all the merge logic locally, which is faster than sending everything to a central server).

You can see that we’re still on the version that’s bolded (“Added Zone 4 to Conveory”). The newer version, on top, is the one with the Nut Feeder change from our friend. In order to merge that change with ours, just right click on their version and click “Merge With…” from the context menu. That will give you a pop-up. It should be telling you, in a long winded fashion, that you’re merging the “other” version into your “local” version. That’s what you always want. Click Merge. It will give you another box with the result of the merge, and in this case it was successful because there were no conflicts. Now click Commit. This actually creates a new version in your repository with both changes, and then updates your local copy to that merged version. Now take a look at the “Line 1.txt” file:

Line 1
  - Conveyor
    - Zone 1
    - Zone 2
    - Zone 3
    - Zone 4
  - Robot
    - Interlocks
    - EOAT
    - Vision System
  - Nut Feeder
    - 120VAC, 6A

It has both changes, cleanly merged into a single file. Cool, right!?

What’s the catch? Well, if the two changes are too close together, it opens a merge tool where it shows you the original version before either change, the file with one change applied, the file with the other change applied, and then a workspace at the bottom where you can choose what you want to do (apply one, the other, both, none, or custom edit it yourself). That can seem tedious, but it happens rarely if the people on your project are working on separate areas most of the time, and the answer of how to merge them is usually pretty obvious. Sometimes you actually have to pick up the phone and ask them what they were doing in that area. Since the alternative is one person overwriting someone else’s changes wholesale, this is clearly better.

Mercurial has a ton of other cool features. You can name your branches different names. For example, I keep a “Release” branch that’s very close to the production code where I can make “emergency” fixes and deploy them quickly, and then I have a “Development” branch where I do major changes that take time to stabilize. I continuously merge the Release branch into the Development branch during development, so that all bug fixes make it into the new major version, but the unstable code in the Development branch doesn’t interfere with the production code until it’s ready. I colour-code these Red and Blue respectively so you can easily see the difference in the repository explorer.

I use the .hgignore file to ignore my active configuration files (like settings.ini, for example). That means I can have my release and development branches in two different folders on my computer, and each one has a different configuration file (for connecting to different databases, or using different file folders for test data). Mercurial doesn’t try to add or merge them.

It even has the ability to do “Push” and “Pull” operations over HTTP, or email. It has a built-in HTTP server so you can turn your computer into an Hg server, and your team can Push or Pull updates to or from your repository.

I hope this is enough to whet your appetite. If you have questions, please email me. Also, you can check out this more in-depth tutorial, though it focuses on the command-line version: hginit.

hg · version-control

In the world of programming there are a lot of PC programmers and comparatively few PLC programmers, but I inhabit a smaller niche. I’m a PLC and a PC programmer. This is a dangerous combination.

If you come from the world of PLC programming, like I did, then you start out writing PC programs that are pretty reliable, but they don’t scale well. I came from an electrical background and I adhered to the Big Design Up Front (BDUF) methodology. The cost of design changes late in the project are so expensive that BDUF is the economical model.

If you come from the world of PC programming, you probably eschew BDUF for the more popular “agile” and/or XP methodologies. If you follow agile principles, your goal is to get minimal working software in front of the customer as soon as possible, and as often as possible, and your keep doing this until you run out of budget. As yet there are no studies that prove Agile is more economical, but it’s generally accepted to be more sane. That’s because of the realization that the customer just doesn’t know what they want until they see what they don’t want.

It would be very difficult to apply agile principles to hardware design, and trying to apply BDUF (and the “waterfall” model) to software design caused the backlash that became Agile.

Being both a PLC and a PC programmer, I sometimes feel caught between these two worlds. People with electrical backgrounds tend to dislike the extra complexity that comes from the layers and layers of abstraction used in PC programming. Diving into a typical “line of business” application today means you’ll need to understand a dizzying array of abstract terminology like “Model”, “View”, “Domain”, “Presenter”, “Controller”, “Factory”, “Decorator”, “Strategy”, “Singleton”, “Repository”, or “Unit Of Work”. People from a PC programming background, however, tend to abhor the redundancy of PLC programs, not to mention the lack of good Separation of Concerns (and for that matter, source control, but I digress).

These two worlds exist separately, but for the same reason: programs are for communicating to other programmers as much as they’re for communicating to machines. The difference is that the reader, in the case of a PLC program, is likely to be someone with only an electrical background. Ladder diagram is the “lingua franca” of the electrical world. Both electricians and electrical engineers can understand it. This includes the guy who happens to be on the night shift at 2 am when your code stops working, and he can understand it well enough to get the machine running again, saving the company thousands of dollars per minute. On the other hand, PC programs are only ever read by other PC programmers.

I’m not really sure how unique my situation is. I’ve had two very different experiences working for two different Control System Integrators. At Patti Engineering, almost every technical employee had an electrical background but were also proficient in PLC, PC, and SQL Server database programming. On the other hand, at JMP Engineering, very few of us could do both, the rest specialized in one side or the other. In fact, I got the feeling that the pure PC programmers believed PLC programming was beneath them, and the people with the electrical backgrounds seemed to think PC programming was boring. As one of the few people who’ve tried both, I can assure you that both of these technical fields are fascinating and challenging. I also believe that innovation happens on the boundaries of well established disciplines, where two fields collide. If I’m right, then both my former employers are well positioned to cash in on the upcoming fusion of data and automation technologies.

TwinCAT 3

I’ve been watching Beckhoff for a while because they’re sitting on an interesting intersection point.

On the one side, they have a huge selection of reasonably priced I/O and drive hardware covering pretty much every fieldbus you’d ever want to connect to. All of their communication technologies are built around EtherCAT, an industrial fieldbus of their own invention that then became an open standard. EtherCAT, for those who haven’t seen it, has two amazing properties: it’s extremely fast, compared with any other fieldbus, and it’s inexpensive, both for the cabling and the chip each device needs to embed for connectivity. It’s faster, better, and cheaper. When that happens, it’s pretty clear the old technologies are going to be obsolete.

On the other side, they’re a PC-based controls company. Their PLC and motion controllers are real-time industrial controllers, but you can run them on commodity PC hardware. As long as PCs continue to become more powerful, Beckhoff’s hardware gets faster, and they get those massive performance boosts for free. Not only that, but they get all the benefits of running their PLC on the same machine as the HMI, or other PC-based services like a local database. As more and more automation cells need industrial PCs anyway, integrators who can deliver a solution that combines the various automation modules on a single industrial PC will be more competitive.

Next year Beckhoff is planning to release TwinCAT 3, a serious upgrade from their existing TwinCAT 2.11. The biggest news (next to support for multiple cores) is that the IDE (integrated development environment) is going to be built around Microsoft’s Visual Studio IDE. That’s a pretty big nod to the PC programmers… yes you can still write in all the IEC-61131-3 languages, like ladder, function block, etc., but you can also write code in C/C++ that gets compiled down and run in the real-time engine.

Though it hasn’t been hyped as much, I’m pretty excited that you can have a single project (technically it’s called a “solution”) that includes both automation programming, and programming in .NET languages like C# or VB.Net. While you can’t write real-time code in the .NET languages, you can communicate between the .NET and real-time parts of your system over the free ADS communication protocol that TwinCAT uses internally. That means your system can now take advantage of tons of functionality in the .NET framework, not to mention the huge amount of 3rd party libraries that can be pulled in. In fact, did you know that Visual Studio has a Code Generation Engine built in? It would be pretty cool to auto-generate automation code, like ladder logic, from templates. You’d get the readability of ladder logic without the tedious copy/paste/search/replace. (Plus, Visual Studio has integrated source control, but I digress…)

Will anyone take advantage?

With such a split between PC and PLC programmers, exactly who is Beckhoff targeting with TwinCAT 3? They’ve always been winners with the OEM market, where the extra learning curve can be offset by lower commodity hardware prices in the long term. I think TwinCAT 3 is going to be a huge win in the OEM market, but I really can’t say where it’s going to land as far as integrators are concerned. Similar to OEMs, I think it’s a good fit for integrators that are product focused because the potential for re-use pays for your ramp-up time quickly.

It’s definitely a good fit for my projects. I’m interested to see how it turns out.

twincat

Talking about voting machines on this blog might seem a little off-topic, but I’m always fascinated by how automation is always interconnected with the people using it. That’s why I think voting machines are fascinating: because people are as much a part of the system as the technology.

I was interested to watch David Bismark’s recent TED talk on “E-voting without fraud”:

The method he’s describing seems to be the same as in this IEEE article.

Now I’m not an expert in the election process, but there are some fundamental things we all understand. One of those fundamental elements is called the “Secret Ballot“. Canada and the US have both had a secret ballot since the late 1800′s. When the concept is introduced in school, we’re shown a picture of how people “used” to cast their votes, which was to stand up in front of everyone at the polling station and call out your choice. Off to the side of the picture, we always saw a gang of people ready to rough up the people who voted for the “wrong” candidate. Therefore, most of us grow up thinking that freedom from retribution is the one and only reason for a secret ballot, so everyone thinks, “as long as nobody can learn who I voted for, then I’m safe.”

That’s really only half the reason for a secret ballot. The other half of the reason is to prevent vote-selling. In order to sell your vote, you have to prove who you voted for. With a secret ballot, you can swear up and down that you voted for Candidate A, but there really is no way for even you to prove who you voted for. That’s a pretty remarkable property of our elections. That’s the reason that lots of places won’t allow you to take a camera or camera-equipped cell phone into the voting booth with you. If the system is working correctly, you shouldn’t be able to prove who you voted for. That means you’re really free to vote for the candidate you really want to win.

I would also like to point out that vote-selling isn’t always straightforward. Spouses (of both genders) sometimes exert extreme pressure over their significant others, and some might insist on seeing proof of who the other voted for. Likewise, while employers could get in hot water, I could easily imagine a situation where proving to your boss that you voted the way he wanted ended up earning you a raise or a promotion over someone who didn’t. All of these pseudo-vote-selling practices always favour the societal group that has a lot of power at the moment, which is why it’s important for our freedoms to limit their influence.

That Means NO Voting Receipts

If you want to design a system that prevents vote-selling, you can’t allow the voter to leave the polling station with any evidence that can be used to prove who they voted for. (The system presented above allows you to leave with a receipt, but they claim it can’t be used to prove who you voted for.)

With this in mind, isn’t it amazing how well our voting system works right now? You mark your ballot in secret, then you fold up the paper, walk out from the booth in plain public view, and you put your single vote into the ballot box with everyone else’s. Once it’s in that box and that box is full of many votes, it’s practically impossible to determine who cast which vote, but if we enforce proper handling of the ballot box, we can all trust that all of the votes were counted.

We Want to Destroy Some Information and Keep Other Information

In order for the system to work correctly, we need to effectively destroy the link between voter and vote, but reliably hang on to the actual vote and make sure it gets counted.

Anyone who has done a lot of work with managing data in computers probably starts to get nervous at that point. In most computer systems, the only way we can really trust our data is to add things like redundancy and audit logs, all of it in separate systems. That means there’s a lot of copying going on, and it’s very easy to share the information that you’re trying to destroy. Once you’ve shared it, what if the other side mishandles it? Trust me, it’s a difficult problem. It’s even more complicated when you realize that even if the voting software was open source, you really can’t prove that a machine hasn’t been tampered with.

The method describe above offers a different approach:

• With the receipt you get, you can prove that it is included in the “posted votes”
• You can prove that the list of “tally votes” corresponds to the list of “posted votes” (so yours is in there somewhere)
• You can’t determine which tally vote corresponds to which posted vote

ATMs and Voting Machines are Two Different Ballgames

One of the things you often hear from voting machine proponents, or just common people who haven’t thought about it much, is that we’ve been using “similar” machines for years that take care of our money (ATMs) and they can obviously be designed securely enough. Certainly if we have security that’s good enough for banks, it ought to be good enough for voting machines, right?

This is a very big fallacy. The only reason you trust an ATM is because every time there’s a bank transaction, it’s always between at least two parties, and each party keeps their own trail of evidence. When you deposit your paycheque into the ATM, you have a pay stub, plus the receipt that the ATM prints out that you can take home with you. On top of that, your employer has a record that they issued you that cheque, and there will be a corresponding record in their bank account statement showing that the money was deducted. If the ATM doesn’t do its job, there are lots of records elsewhere held by third parties that prove that it’s wrong. An ATM is a “black box”, but it has verifiable inputs and outputs.

The system above attempts to make the inputs and outputs of the voting system verifiable.

Another Workable E-voting System

The unfortunate thing about the proposed system, above, is that it’s rather complicated. If you read the PDF I linked to, you need a couple of Ph.D. dissertations under your belt before you can make it through. I don’t like to criticize without offering a workable alternative, so here goes:

Paper Ballots

If you want to make a secret ballot voting system that’s resistant to fraud, you absolutely need to record the information on a physical record. If you want to make it trustworthy, the storage medium needs to be human readable. Paper always has been, and continues to be, a great medium for storing human readable information in a trustworthy and secure way. There are ways to store data securely electronically, but at the moment it requires you to understand a lot of advanced mathematical concepts, so it’s better if we stick with a storage medium that everyone understands and trusts. In this system we will stick with paper ballots. They need to go into a box, in public view, and they need to be handled correctly.

Standardized Human and Machine Readable Ballots

Some standards organization needs to come up with an international standard for paper ballots. This standard needs to include both human and machine readable copies of the data. I suggest using some kind of 2D barcode technology to store the machine readable information in the upper right corner. Importantly: the human readable and machine readable portions should contain precisely the same information.

Please realize I’m not talking about standardized ballots that people then fill out with a pencil. I’m talking about paper ballots that are generated by a voting machine after the voter selects their choice using the machine. The voter gets to see their generated paper ballot and can verify the human readable portion of it before they put it into the ballot box.

Voting Machines vs. Vote Tallying Machines

Now that we have a standardized ballot, the election agencies are free to purchase machines from any vendor, as long as they comply with the standard. There will actually be two types of machines: voting machines that actually let the voter generate a ballot, and vote tallying machines that can process printed ballots quickly by using the machine readable information on each ballot.

One of the goals of e-voting is to be able to produce a preliminary result as soon as voting has completed. Nothing says that the Voting Machines can’t keep a tally of votes, and upload those preliminary results to a central station when the election is complete. However, the “real” votes are the ones on paper in the ballot boxes.

Shortly after the election, the ballot boxes need to be properly transported to a vote tallying facility where they can be counted using the vote tallying machines, to verify the result.

Checks and Balances

Part of the verification process should be to take a random sample of ballot boxes and count them manually, using the human readable information, and compare that with the results from the vote tallying machine. This must be a public process. If a discrepancy is found, you can easily determine if it was the voting machine or the vote tallying machine that was wrong. Assuming the ballots were visually inspected by the voters, then we can assume that the human readable portion is correct. If the machine readable information doesn’t match the human readable information, then the voting machine is fraudulent or tampered with. If the machine and human readable information match, then the vote tallying machine is fraudulent or tampered with.

If one company supplied both the voting machines and the vote tallying machines, then it would be a little bit easier to commit fraud, because if they both disagreed in the same way, it might not be caught. That’s why it’s important that the machines are sourced from different independent vendors.

No Silver Bullet

Notice that none of the current or proposed solutions are successfully resistant to someone taking some kind of recording equipment like a camera or a cell phone with camera into the voting booth with them. We still need some way to deal with this.

voting

When I first saw the Model-View-ViewModel pattern, I thought it was pretty cool. I actually wrote an entire framework and an application using the MVVM pattern. It works, and it gives you a nice separation of concerns between your Model and the rest of your application.

What’s never sat well with me is the amount of redundant and sometimes boilerplate code you have to write in your ViewModel. Assuming you have POCO objects in your domain model and that your domain model shouldn’t know anything about your ViewModel (it shouldn’t), then if you have a domain class called Customer with a Name property, chances are you’ll have a ViewModel class called Customer, with a property called Name, except that the ViewModel will implement INotifyPropertyChanged, etc. It works, but once you get down to coding, it’s a LOT of extra work.

There is an alternative out there called Model-View-Presenter. Most people claim that MVVM is a form of MVP, but once you look closely, that’s not the case (or at least, that’s now how people are using MVVM). In both MVP and MVVM architecture, the ViewModel/Presenter forms a separation between the View and Model. The difference is that in MVVM, the ViewModel works with Model objects explicitly, but in MVP both the View and the Model are abstracted services to the Presenter. Perhaps it’s clearer with an example:

class CustomerViewModel
{
    Customer wrappedCustomerModel;
    public CustomerViewModel(Customer customerToWrap)
    {
        wrappedCustomerModel = customerToWrap;
    }

    // Leaving out the INotifyPropertyChanged stuff
    public string Name
    { get { return wrappedCustomerModel.Name; } }
}

class Presenter
{
    public Presenter(IView view, IModel model)
    {
        populateViewWithModelData(view, model);
        view.UserActionEvent +=
            new UserActionEventHandler((s, e) =>
            {
                model.ProcessAction(e.Action);
                populateViewWithModelData(view, model);
            });
    }
    private void populateViewWithModelData(
        IView view, IModel model)
    {
        // custom mapping logic here
    }
}

There’s at least one major benefit to the Presenter class over the ViewModel class: you can wrap the model.ProcessAction class in a try...catch block and catch all unhandled exceptions from the Model logic in one place, logging them, and notifying the user is a nice friendly way. In the ViewModel case, any property getter can throw an exception, which causes lots of problems in WPF, not the least of which is that it sometimes breaks the binding and no further updates get sent back and forth.

Now let’s look at the constructor of the Presenter again:

    public Presenter(IView view, IModel model)

Nothing says that the View the presenter is hooking into couldn’t be a ViewModel:

    public Presenter(IViewModel viewModel, IModel model)

If you do this, then the Presenter separates the ViewModel from the Model! Ok, does that sound like too much architecture? Why did we want a ViewModel in the first place? We wanted it because we wanted to make the GUI logic testable, and then use WPF’s binding mechanisms to do a really simple mapping of View (screen controls) to ViewModel (regular objects). You still get that advantage. You can create a ViewModel that implements INotifyPropertyChanged and fires off an event when one of its properties changes, but it can just be a dumb ViewModel. It becomes a “Model of the View”, which is what the ViewModel is supposed to be. Since the ViewModel then has no dependencies on the Model, you can easily instantiate mock ViewModel objects in Expression Blend and pump all the test data you want into them.

Doesn’t that mean we’ve shifted the problem from the ViewModel to the Presenter? The Presenter obviously has to know the mapping between the Model and the ViewModel. Does that mean it reads the Customer Name from the Model and writes it into the Customer Name property in the ViewModel? What have we gained?

What if the Presenter was smart? Let’s assume that IModel represents that state of some domain process the user is executing. Maybe it has a Save method, maybe an Abort method. Perhaps it has a property called CustomerAddress of type Address. Maybe it has a read-only property of type DiscountModel, an Enum. Even though we’re working against an abstract IModel interface which probably doesn’t include all of the concrete public properties and methods, we have the power of reflection to inspect the actual Model object.

What if the presenter actually generated a new AddressViewModel and populated it with data from the Model any time it saw a public property of type Address on the concrete Model object? What if it hooked up listener events, or passed in a callback to the AddressViewModel so it could be notified when the user modified the address, and it would write those changes back to the Model, then inspect the Model for changes and update the ViewModel with the results? What if when it saw an Enum property on the Model, it automatically generated a DropDownListViewModel? What if, when it sees a Save method, it generates a SaveViewModel that gets mapped to a button in the View?

Can we write a generic Presenter that can comprehend our Model and ViewModel objects? Can it even build the ViewModel for us, based on what the concrete Model object looks like, and perhaps based on some hints in a builder object that we pass in?

The answer to all these questions are “Yes.” We can use the Presenter to automate the generation of the ViewModel layer based on the look & feel of the domain model itself. I leave this as an exercise for the reader…

mvp · mvvm

I’ve had my head down working on SoapBox Snap recently (an open source, free ladder logic editor and runtime for your PC), so I decided it was a good time to come up for air and write a blog post. A lot has happened with Snap since I posted the sneak peek back in July. I’ve flushed out a good ladder logic instruction set, online debugging is working, and you can now execute the runtime as a Windows service, so it’ll keep running your logic in the background, and even auto-start when Windows starts.

It’s been a long time to get a first version out the door, but it’s always been the plan to adopt an agile workflow after release. That is, short release cycles and continuous small improvements. In fact, that’s what I’m going to talk about… why not to use agile releases during the initial development.

The image above is currently my September calendar picture at work, which made me think of this. Please click on the image to go to Despair Inc. and take a look at their stuff. It’s hilarious.

Writing SoapBox Snap took a lot of planning and design:

I started by tackling online programming. Downloading the entire application every time there’s a change won’t scale as the program grows. That is, how do you design a data structure such that I can modify it locally, generate a packet of data that only contains the difference between this version and that last version, transfer that change over a communication channel, and reconstruct the new version on the other end given the previous version and the difference? I created a library for building this type of data structure, and the communication protocols to make it work, and I packaged it in a library called SoapBox.Protocol.Base. Then I build a data structure for automation programs on top of that and put it in a library called SoapBox.Protocol.Automation. If you follow standard software architecture terminology, I now had my “Model”.

Then I decided to tackle how to make an extensible editor and runtime. I wanted other people to be able to extend SoapBox Snap with new ladder instructions and other features, so extensibility had to be built-in from the ground up. After looking at the various technologies, I settled on .NET’s new Managed Extensibility Framework (MEF), which was only recently released in .NET 4. After playing with it for a while, I realized that part of what I was building was applicable to anyone making an editor-like application with extension points. I decided to encapsulate the “framework” part of it into a re-usable library called SoapBox.Core, and I released it as open source and posted an article on CodeProject about how to use it. Over several months, people have started downloading, using, and even contributing changes back to SoapBox Core to improve it. We’ve setup a Q&A site for people to ask questions, get help, and give feedback.

Armed with a Model, and a Framework, I set off to build SoapBox Snap. At times I made some wrong turns, or started down some dead-ends, but I had a good vision of what I wanted to build. There were nights where I went to bed feeling like I’d banged my head against the keyboard for a few hours and accomplished nothing, but every morning brought a fresh perspective, idea, or insight that helped move the project forward. I didn’t accept any compromises on the core features that affect everything else, like undo/redo. If you don’t get undo/redo right at first, adding it in later means major architectural upheaval.

Ironically, the first problem I solved at a base level, online programming, is only half-implemented in this first version. However, since everything is already there to support full online programming, no major architectural changes have to be made to add it later. My approach was decidedly “bottom-up”. Agile is “top-down”. Would Agile have worked better for this project, or was I right to take a bottom-up approach?

I believe Agile has one failure point: interoperability (and SoapBox Snap is all about interoperability). If you have one closely-knit team doing the development and they’re the only ones that will ever interact with the edges of this application, then I think Agile works well. On the other hand, when you have extensibility points, API’s, or common file formats that 3rd parties are depending on, then doing the kind of massive refactoring that’s required to iteratively change a one-month-old barely-working application into a fully developed one is either going to break the contracts with all 3rd parties, or you’re going to have to support broken legacy interfaces for the rest of your application life cycle. Spending the extra time to build your application bottom-up, and releasing a relatively stable architecture to 3rd parties with working extensions and file formats greatly reduces the friction that Agile development would have caused.

At any rate, we’ve waited long enough. It’s almost over: look for it to be released in early October. I can’t wait to see what people will do with it.

agile · snap

I’ve been playing around with F# recently, the functional language that shipped with Visual Studio 2010. I’m looking at using it to write an application using WPF and the Model-View-ViewModel architecture. One big requirement is for DataBinding.

When you bind the View to the ViewModel, you typically have to use the explicit name of the property on the ViewModel that you’re binding to (like “Text”). You also need the literal name of the property when you fire off (or receive) the PropertyChanged event. That’s always been a little ugly, because using the literal string means it isn’t compile-time checked. I got around it in C# using a helper class which uses reflection and lambda expressions to look at a piece of code (e.g. o => o.MyProperty) and get the name of the property as a string.

That utility class didn’t work in F#, mostly because F# lambda expressions aren’t the same base object as C# lambda expressions. I was faced with rewriting it. This is where F# seems to shine. Here’s the same “get property name” logic written in F#:

    open Microsoft.FSharp.Quotations.Patterns

    let propertyName quotation =
        match quotation with
        | PropertyGet (_,propertyInfo,_) -> propertyInfo.Name
        | _ -> ""

Here’s how you can use it:

    type myClass(p) =
        member x.MyProperty
            with get() = p

    let myObject = new myClass(1)

    let myPropertyName = propertyName <@ myObject.MyProperty @>

At the end, myPropertyName has been assigned the string value “MyProperty”. It’s a heck of a lot less code. In this case it only works if you have an existing object to run it against. However, you can modify the propertyName function to make it recursively dig through the Lambda and find the PropertyGet:

    let rec propertyName quotation =
        match quotation with
        | PropertyGet (_,propertyInfo,_) -> propertyInfo.Name
        | Lambda (_,expr) -> propertyName expr
        | _ -> ""

    let myPropertyName = propertyName <@ fun (x : myClass) -> x.MyProperty @>

Now you don’t need to have an instance of the class lying around to get the property name.

No tags

This week I did something I’ve probably done a hundred times before, but this time it felt absolutely absurd. It’ll probably be the last time I ever do it.

What was this crazy event? I purchased shrink-wrapped software. It was an upgrade copy of Visual Studio 2010 (upgrade from 2008 Standard Edition).

I had to add a cute little piggybank because otherwise it just looked pathetic.

This little box of bits says it was “made in Puerto Rico”. I guess that means the RTM build was FTP’d to some server in Puerto Rico where it was burned onto a dual layer DVD. In fact it’s likely the DVD’s were burned somewhere else and shipped to Puerto Rico, but anyway, this DVD was then stuffed into plastic case along with a 5×7 piece of heavy stock paper with a beat up yellow sticker on it: the license key.

This plastic case was then stuffed into a cardboard sleeve and shrink-wrapped. It probably made its way to a distribution center before being sold to an online distributor CDW, where it sat on a shelf for a few weeks.

It was at this time that I started hunting around for a Visual Studio 2010 license. I had a copy of VS 2008 and I knew there was a discount for upgrades. You can actually download fully licensed copies from the Microsoft store for $399 (CDN), but I found this boxed copy for about $30 cheaper. Of course I still had to pay S&H, but that was about $13 (UPS Ground).

That was on Friday. Today it’s Monday and I received a call from the local UPS distributor. They’d put it on the wrong delivery truck today, so they wouldn’t be able to deliver it until tomorrow. I wasn’t going to be home anyway, so I told them I’d pick it up tonight after work.

I drove a good 25 km out of my way to go pick it up. The truck wasn’t back from its run yet when I got there, so I stood in line another 20 minutes waiting. I’m not knocking UPS here: their whole system boggles my mind. They knew exactly where this package was 100% of the time, even when it ended up being loaded on the wrong truck, and it was in my hand about 20 seconds after the truck pulled up to the building.

Never underestimate the bandwidth of a UPS truck full of dual layer DVDs.

After installing the software, the first thing it asks you to do is check for updates online. That’s because the version that came on the disk was probably out-of-date before it made it out of Puerto Rico. The only thing I really bought was a yellow sticker with my 16 digit license key on it. I could have downloaded a fully working copy last Friday night, and had it in about 2 hours. It would have worked for 60 days, and you could extend it for another 60 for no cost at all. The only thing of value was the legal right to use this software past the evaluation date. The 16 digit license key is only a proof of purchase.

The absurdity of shipping useless plastic and paper all of the continent, driving out of my way and even standing in line to pick it up, just to “prove” that I paid for a legal license to use the software — it’s really striking isn’t it? What’s crazier is how normal this seemed ten or even five years ago!

It’s not like I wouldn’t have purchased the $399 copy from Microsoft if that’s the only one I could find. I went through this because, well, given the chance to save about $15, I’m just cheap. I guess they figure some people won’t bother to pay money for something they can’t hold in their hand, but aren’t we past that now? Apple figured it out. Look at the iPhone App Store, and iTunes.

If I relate this story to my daughter ten years from now, she’ll think I’m nuts. You bought software how? Why?!?

Well, good riddance, shrink-wrapped software. Rest in peace.

shrinkwrap

The “low intellectual property” Fashion Industry as a model for innovation?

I have to agree. Since I’ve started getting involved in open source software, I can tell you that intellectual property laws are the biggest obstacle to innovation that I face. I can say that because I took the time to consult with intellectual property lawyers on everything from trademark to copyright to open source license to software patent. Here’s what I found out: you can’t write a single line of code without violating someone’s patent, somewhere. However, the only time that person will enforce their patent claim is if you’re really successful. The only defense is to pay tens or hundreds of thousands of dollars to build up a patent portfolio of your own so you can partake in a ludicrous arms-race deterrent. On top of that, your patent portfolio offers no defense against so-called “patent trolls”: companies that only own patents and don’t write software of their own. They’re the equivalent of a terrorist cell with nuclear weapons… your nukes are ineffective because you don’t have a city to target.

Copyright is under control, mostly because it’s so much weaker than patents, and it’s free. It stops whole-sale copying of your software, but allows someone to look at your source code, learn from it, and then write it themselves without violating copyright (if they’re careful). Plus, the wide adoption of open source licenses give us lots of material that we can freely copy or integrate.

The irony is that patents were supposed to speed up innovation, but they’ve done exactly the opposite in the software industry.

Do you want to know why none of this matters though? I’m going to make a bold prognostication here:

If the last decade (have we agreed on what to call it yet? The Oh’s?) was about more crap, then the next decade will be about customized crap. Seriously, we gorged ourselves on the mountain of crappage that is sold at Wal-mart and people are starting to wake up with a hangover. We’ve already seen the desires start to change from more to better but I think we’re going to see it change to mine. People are going to want unique stuff. Things that they had a hand in customizing. Something they can use to express themselves.

This has major implications for manufacturing, so it affects the automation industry, and I think it also affects software. Just like the fashion industry, where people use clothes to express themselves, we’re about to enter an age when all the crap you sell to consumers has to have a story, a personality, and be unique. Mass production will remain for your basic staples, and China is going to continue to provide us with that, but North America and Europe are going to have to start producing bespoke crap.

What does this mean for automation? We’re going to see a rise in the build-a-bear style build-your-own-product-on-a-website and have it manufactured and sent to your door the next day. Imagine the logistical changes that need to take place to make that happen. Internet merchandising companies like cafepress already have a leg up in this department. Their merchandise is manufactured after you order it. Now imagine a product with 10,000 or a million different variations, and the automation that has to support it. Imagine the automation required to manufacture products whose specifications change as fast as fashion trends.

On the software side, we’re going to see every major application support custom 3rd party add-ins so everyone can customize their software so it works for them. I think we’re also going to see a rise in “pseudo-programming languages” that let people who aren’t programmers actually customize their applications in ways that we’ve never let them do before. Platforms that give people the power to build solutions to their own needs are going to flourish, and one-size-fits-all solutions like word processors will fade into obscurity.

So I think we’ll have two options: the vast majority of us are going to spend our time building custom things for individuals or businesses. The successful ones will produce products that people can customize themselves.

deep-thoughts · innovation · plc

I spend a significant amount of my time these days doing PC programming (as opposed to PLC programming) and I’d say the most time consuming part of my job is following the DRY Principle, aka “Don’t Repeat Yourself”.

The naive interpretation of the DRY principle is that it’s about minimizing the amount of code you write. While this may sometimes be a side effect, this isn’t the point of DRY at all. What we’re trying to do is structure the program so that there’s just one place for everything, and everything is in it’s one place. This is a lot harder than it sounds because no matter how you segment your program into modules, classes, or layers there’s always something that cuts across those boundaries.

Take logging for instance. Every part of the application needs to log things to the same log file. The naive implementation of opening a file, writing a line, and closing it is short but doesn’t follow the DRY principle because we’re repeating this simple sequence of steps all over the place. The danger is that if we change our logging strategy, we now have to change multiple places in every file.

There are probably two ways that the “logging strategy” could change:

1. We change where (or if) we want to log the data, like to a database instead of a file
2. We change the kinds of stuff we want to log

We can handle the first case easily by moving the logging logic into a global subroutine, or if you’re more advanced you would use something like a logging service with the service locator pattern. That would let us change where we log the data, and if we include a severity level with the logging interface, we could filter our logging by severity.

The second case is really hard to solve. We have to choose each spot in the code where we want to log information. Maybe every time I catch an error in a try/catch block, but that means every time I write a try/catch block I have to add a call to my logging service. This case violates the DRY principle because the idea is just “every time I catch an error in a try/catch block”. That idea needs to be codified in one place. Ideally I should be able to extend the try/catch block of the language and say, “any time you execute a catch I also want you to do this”. That’s not possible with .NET, that I know of anyway.

Likewise, maybe I want to know every time a user carries out any action that might affect the underlying data in the database, and I’ll have to add logging code at each of those places too. If you have all of your database access going through a single layer in your application, at least you can confine the logging to that layer, but it may consist of dozens of classes representing dozens of database tables. That’s where we need something like the ADO.NET entity framework, and have all of our database entities derive from a single base class, and then we might be able to tie our logging rules in there.

The problem is that these architectural solutions are hard. They take a lot of effort, and when you just need to add logging, it’s too easy to just go through your code and add logging wherever you need it, and you’ve created a maintenance nightmare. It’s the extremely low cost of copy and paste vs. architecture that creates this problem. If you had to pay $1 every time you copied and pasted code, I bet you’d write more maintainable software.

.net · dry-principle

I got ahold of some Phidgets on the weekend. These are basically USB I/O devices for amateur robotics, but I’m looking at their use in the home automation space.

They have drivers for lots of operating systems, and APIs for almost any programming platform under the sun.

Getting the I/O connected and controlled from a .NET application was a breeze, including the Interface Kit 8/8/8 and the little R/C Servo Controller. That little servo could certainly move a damper in a heating duct, and they have lots of environmental sensors. The wheels have started turning…

.net · phidgets