Long silence

It's been over a month since my last post - I'm sorry about this. I have a horrible tendency to forget my blogs, as my MySpace friends can attest to. I have an excuse this time though - my workload has effectively doubled at the office, and I've had to budget my time recently.

I also spent some time down at my company's warehouse getting to know the warehouse staff and seeing the physical processes that drive the company. It was definitely an eye-opening experience - this was the first time I've ever visited a warehouse of any sort, and seeing them physically carry out the processes that I've helped to develop was gratifying in a way. It helped me to fully understand the ramifications of the decisions that I make when building the software that they'll use.

My spare time (hah!) has been spent getting the new bot up and running - so far, so good! :) I have a command-line version up and running that is fully configurable and about 70% functional - no hunting functionality is in, but harvesting works a charm. I've opened up beta applications on the Macrocrafter website, so you should hop over there if you're interested in helping me test.

The development process has been much easier this time around, I think. I wasn't very smart with the first few iterations of the harvest bot - during the alpha and beta periods I pretty much wrote EverHarvest by the seat of my pants, largely without a plan at all. The result was that after the command-line tests were complete, I had to completely rewrite the system because I had coupled it into the command-line project. This time, I'm developing the engine of the system completely separate from the UI - it's in its own separate assembly - which will make it *much* easier to move from the command-line interface to the graphical interface.

So these are the developments of late. Nothing earth-shattering, but lots of things to distract me from blogging. I'll try to keep up moving forward. =)

I keep learning new things about C#

Apparently, in the body of a method, you can arbitrarily use braces to create 'pockets' of child-local scope. For example:

static void Main(string[] args) {

    int x = 10;

 

    {

        int y = 1;

        Console.WriteLine(x + y);

    }

 

    {

        int y = 2;

        Console.WriteLine(x + y);

    }

 

    Console.ReadLine();

}

This will, as expected, create the output 11 and 12. The variable 'y' is created twice, but in each case it's created in some sort of anonymous child scope. I was always under the impression that you had to have a reason of some sort in order to create child scope like this. Glad to see I was mistaken - this might come in handy some day...

Progress

To cut to the chase - progress on the new harvest/hunter bot is going very slow. Obviously I can't work on it while at work - sort of a conflict of interest when I'm supposed to be working on work stuff while at work - so my availability is limited. Weekends are limited as well, since because I work a 'real job' I only get to really see the kids during the weekends, and because the last thing I want to do during my days off is work...

More technically, I'm blocked on a couple issues. Since I'm merging a harvest bot and a hunter bot together, there are some technical difficulties to work out - they have to share targeting systems, navigation systems, and some settings which, from a UI perspective, will have to be separated yet united somehow... As far as the engine itself goes - well, because I'm using my workflow system, it's actually not going to be as hard as it seems.

I've been using the workflows as logical separations between the various systems involved. One workflow for hunter, one for harvester, and one for navigation. Targeting is too simple a process for an entire workflow - a single workitem will be sufficient. Finally, to tie everything together, a master workflow responsible for coordinating the efforts of these disparate (yet united) tasks.

The master workflow will be responsible for invoking the navigation sub-workflow, as well as the harvester and hunter workflows. All workitems for these sub-workflows are yielded to the workflow runner, so the entire hierarchy of workflows is flattened to a sequential list as its run - as it should be.

Where I'm being held up is in how to handle tasks that should be shared by all these workflows, but must happen separately in each of them. For example, stuck checking / handling. This sounds like a navigation concern, and it is, but it's also a concern for harvesting and hunting - in particular, moving from the walkpath to a node / mob, and moving back to the walkpath. Both of those tasks are prone to dealing with the stuck issue, so I'll need to have stuck checking there as well. Should I create another workflow - a stuck check workflow - that's responsible for detecting and dealing with the issue? If so, how do I invoke it while ensuring the harvest, hunter, and navigation workflows are ignorant of it? That's one of the goals for workflows - keeping the sub-workflows ignorant to each other and letting the master workflow do the coordination.

Maybe that is the answer. I can inspect the workitems given to me from the Hunter and Harvest workflows before I yield them - maybe I can check the specifics, and if they're in their navigation phases I can interleave the stuck checking in there with them. Hah - yes, I'll do it. =) Sometimes just talking through a problem can lead you to a solution - I'll give it a try when I get home.

Well, anyway - I need to get back to work. Lunch break only lasts so long...

Little accomplishments

It's great when I get to put new things I've learned to work, especially when it's to the greater good.

Case in point - did you know that assignment operations return the result of the assignment? I didn't until recently - it's one of those obscure language bits that almost never come up, but when they do they can really make life easier.

I'm working on my workflow idea (see my previous post for details) - in particular, I'm toying with the idea of composing workflows from sub-workflows - and so I wrote a workflow that takes two workflows and interleaves their workitems together. At first I used a 'Zip' method which does the job, but if there are more workitems in one workflow than the other, the extras are truncated - this is not what I wanted; I want the extra workitems returned correctly.

So I wrote this Workflow() method, which does the job:

do {

    if (enum1.MoveNext()) {

        yield return enum1.Current;

    } else {

        enum1Done = true;

    }

 

    if (enum2.MoveNext()) {

        yield return enum2.Current;

    } else {

        enum2Done = true;

    }

} while (!(enum1Done && enum2Done));

Interleaving workflow that yields extras

It works, but is verbose and ugly. It is very much imperative code - the *how* of the code obscures much of the *what* and the *why*.

Obscure language features come to the rescue, however, when a little light-bulb goes off in my head. Because assignment operators return the result of the assignment, I can combine the assignment and the test in one statement. The original code becomes this:

var enum1Alive = false;

var enum2Alive = false;

while ((enum1Alive = enum1.MoveNext()) | (enum2Alive = enum2.MoveNext())) {

    if (enum1Alive) yield return enum1.Current;

    if (enum2Alive) yield return enum2.Current;

}

From 14 LOC to 6 - good stuff

This may seem like child's play to some of you, and you're probably right, but for me, this represents an accomplishment (albeit small) in my understanding of the language I use.

Yield, and the C# state machine

For those of you who don't already know, the C# compiler has its own state machine generator that you can use. It's true - it's called the 'yield' statement!

Yield is used to create implementations of the Enumerable pattern - a software pattern that allows you to treat a collection of things as an enumeration, over which you can perform some process. In C#, you consume an enumeration via the 'foreach' statement, like so:

    1 IEnumerable<string> ies = new List<string>() { "asd", "ert", "qwe", "fgh" };

    2 

    3 foreach (var s in ies) {

    4     Console.WriteLine(s);

    5 }

Iterating over a List of strings

Before C# 2.0, creating a custom enumerable meant implementing the Enumerable Pattern via IEnumerator and IEnumerable. I don't feel like going through this, so here's a short and sweet example I found online.

This is a fairly common implementation. In fact, it is so common that when the C# language designers were conceiving 2.0 of their product, they chose to make it a first-class compiler-driven feature. Enter the yield keyword, which is capable of turning the example above into the following code:

    1 public IEnumerable<char> MyChars() {

    2     yield return 'A';

    3     yield return 'B';

    4     yield return 'C';

    5     yield return 'D';

    6 }

Implementing IEnumerable and IEnumerator via the yield keyword

This is much more straightforward, but there is some magic happening here that allows this to happen. First of all, the Enumerable pattern includes the requirement that the processing of the enumeration be lazy - that is, evaluated on a need basis. This allows an enumeration to contain an effectively infinite amount of items. Such an enumeration can be created using this code:

    1 public IEnumerable<bool> infiniteAlternatingBools() {

    2     bool cur = false;

    3     while (true) {

    4         yield return cur;

    5         cur = !cur;

    6     }

    7 }

Creating an enumeration of an infinite pattern of alternating booleans

This code generates a list of alternating booleans - True / False / True / False - forever. Surely, this code will result in a locked-up process. Not so, fortunately for us, because behind the scenes (in the magic part) this code is expanded into a proper implementation of the Enumerable pattern - lazy evaluation included. Only when you request the next value is it generated and then provided, meaning this infinite generation can be short-circuited at any moment.

How? Magic, like I said - although this magic can be explained through gratuitous use of .NET Reflector. According to .NET Reflector, my infiniteAlternatingBools() method looks like this:

    1 public IEnumerable<bool> infiniteAlternatingBools()

    2 {

    3     <infiniteAlternatingBools>d__5 d__ = new <infiniteAlternatingBools>d__5(-2);

    4     d__.<>4__this = this;

    5     return d__;

    6 }

Reflector output of infiniteAlternatingBools method

What? This is a mess. What is reflector telling me about this code?

In a nutshell, Reflector is saying that the C# compiler has, behind my back, taken the code I wrote and moved it into an anonymous private class. The constructor of that class takes an integer in its constructor, which the rewritten method initializes to -2. It also sets a 'this' property to the class that contains infiniteAlternatingBools - probably allowing it to access the original class's private members. Then the rewritten method returns the instance of that anonymous class - which suggests that it implements IEnumerable<bool>.

Kind of rude, don't you think? Replacing our carefully written infinite loop with some object creation? Actually the C# compiler has done us a favor - if you look in the anonymous class it generated you'll find the original code you wrote, albeit in a form you might not fully recognize. Here's the listing of the class (cleaned up a little bit from the Reflector version, which contains illegal characters):

    1 [CompilerGenerated]

    2 private sealed class d__5 :

    3     IEnumerable<bool>, IEnumerable,

    4     IEnumerator<bool>, IEnumerator,

    5     IDisposable {

    6 

    7     // Fields

    8     private int state;

    9     private bool current;

   10     public Program.anon _this;

   11     private int initialThreadId;

   12     public bool _6;

   13 

   14     // Methods

   15     [DebuggerHidden]

   16     public d__5(int state) {

   17         this.state = state;

   18         this.initialThreadId = Thread.CurrentThread.ManagedThreadId;

   19     }

   20 

   21     public bool MoveNext() {

   22         switch (this.state) {

   23             case 0:

   24                 this.state = -1;

   25                 this._6 = false;

   26                 break;

   27 

   28             case 1:

   29                 this.state = -1;

   30                 this._6 = !this._6;

   31                 break;

   32 

   33             default:

   34                 return false;

   35         }

   36         this.current = this._6;

   37         this.state = 1;

   38         return true;

   39     }

   40 

   41     [DebuggerHidden]

   42     IEnumerator<bool> IEnumerable<bool>.GetEnumerator() {

   43         if ((Thread.CurrentThread.ManagedThreadId == this.initialThreadId) && (this.state == -2)) {

   44             this.state = 0;

   45             return this;

   46         }

   47         Program.anon.d__5 d__ = new Program.anon.d__5(0);

   48         d__._this = this._this;

   49         return d__;

   50     }

   51 

   52     [DebuggerHidden]

   53     IEnumerator IEnumerable.GetEnumerator() {

   54         return this;

   55     }

   56 

   57     [DebuggerHidden]

   58     void IEnumerator.Reset() {

   59         throw new NotSupportedException();

   60     }

   61 

   62     void IDisposable.Dispose() {

   63     }

   64 

   65     // Properties

   66     bool IEnumerator<bool>.Current {

   67         [DebuggerHidden]

   68         get {

   69             return this.current;

   70         }

   71     }

   72 

   73     object IEnumerator.Current {

   74         [DebuggerHidden]

   75         get {

   76             return this.current;

   77         }

   78     }

   79 }

Listing of the generated anonymous class implementing the Enumerable pattern

This listing is a bit hard to understand. There are fields called state, current, _this, initialThreadID, and _6. There are the IEnumerable and IEnumerator implementations - MoveNext, Current, GetEnumerator, and Reset. There's a constructor (taking an int). What can all this mean? More importantly, where's my infinite loop?

There is no infinite loop. My code is still here, but it's been turned into a state machine. The value that gets passed in to the constructor (-2) tells this state machine that it's in the initial state. When GetEnumerator is called, it checks to see if it's in its initial state - if it's not, it creates a new version of itself and returns that - but if it is, then it moves into state '0' and returns itself. When MoveNext is called, it uses the state to determine what value to set as the 'current' property. At state 0, the initial value is returned - which the C# compiler correctly determined to be false, given my initial 'bool cur = false;' statement. It also moves into state 1. Subsequent calls to MoveNext will call my code which alternates this _6 value, which is a boolean, between true and false - mimicking the behavior I coded.

My infinite loop turned into a lazily evaluated Enumerable Pattern implementation which uses a state machine to decide on what the 'current' value should be whenever MoveNext is called. pretty damned cool if you ask me.

The coolest thing about this is that you can use any C# constructs you want in your enumerable, and create some incredibly complex generators. In my case, I'm taking advantage of the built-in state machine to create a workflow-like process. One place I plan on using this is in EverHarvest 2. Here's a simplified version of what my workflow might look like once fully implemented:

    1 public IEnumerable<WorkUnit> Workflow() {

    2     yield return new Initialize();

    3 

    4     while (true) {

    5         var wp = GetNextWaypoint();

    6         var wtw = new WalkingToWaypoint(wp);

    7         while (!wtw.ReachedWaypoint) {

    8             yield return wtw;

    9 

   10             var tgt = new Targetting();

   11             yield return tgt;

   12 

   13             if (tgt.FoundTarget) {

   14                 if (IsNode(tgt.TargetName)) {

   15                     var wtn = new WalkingToHarvestable(tgt.TargetName, tgt.TargetLocation);

   16                     while (!wtn.ReachedNode) {

   17                         yield return wtn;

   18                     }

   19 

   20                     var h = new Harvesting();

   21                     while (!h.DoneHarvesting) {

   22                         yield return h;

   23                     }

   24                 }

   25             }

   26         }

   27     }

   28 }

EverHarvest Workflow example

Notice how simple this is to understand. It reads very procedurally, and yet because this is lazily evaluated, this process can be interrupted at any of the yield points. This control lies with the code that is enumerating through the workflow - that code can act as the gatekeeper, deciding when to get the next work item, when to execute it, when to break out of the loop, what data each bit should have, etc. This can be done in a foreach statement, or I can use the older MoveNext / Current members.

I hope this helps those of you who are still reading to understand how the yield statement can be used to take advantage of the state machine functionality that the C# compiler provides for us. In terms of readability and maintenance, it has proven to be a real boon for me. I hope this has helped you to find the same benefit.


Edit: Here's the disassembled MoveNext() method from Reflector - I haven't cleaned it up a bit. Lots of red squiglies in this one...

    1 private bool MoveNext()

    2 {

    3     bool CS$4$0002;

    4     switch (this.<>1__state)

    5     {

    6         case 0:

    7             this.<>1__state = -1;

    8             this.<>2__current = new Initialize();

    9             this.<>1__state = 1;

   10             return true;

   11 

   12         case 1:

   13             this.<>1__state = -1;

   14             goto Label_01CB;

   15 

   16         case 2:

   17             goto Label_00B4;

   18 

   19         case 3:

   20             goto Label_00E0;

   21 

   22         case 4:

   23             goto Label_0159;

   24 

   25         case 5:

   26             goto Label_0198;

   27 

   28         default:

   29             return false;

   30     }

   31 Label_01CB:

   32     CS$4$0002 = true;

   33     this.<wp>5__1 = this.<>4__this.GetNextWaypoint();

   34     this.<wtw>5__2 = new WalkingToWaypoint(this.<wp>5__1);

   35     while (!this.<wtw>5__2.ReachedWaypoint)

   36     {

   37         this.<>2__current = this.<wtw>5__2;

   38         this.<>1__state = 2;

   39         return true;

   40     Label_00B4:

   41         this.<>1__state = -1;

   42         this.<tgt>5__3 = new Targetting();

   43         this.<>2__current = this.<tgt>5__3;

   44         this.<>1__state = 3;

   45         return true;

   46     Label_00E0:

   47         this.<>1__state = -1;

   48         if (this.<tgt>5__3.FoundTarget && this.<>4__this.IsNode(this.<tgt>5__3.TargetName))

   49         {

   50             this.<wtn>5__4 = new WalkingToHarvestable(this.<tgt>5__3.TargetName, this.<tgt>5__3.TargetLocation);

   51             while (!this.<wtn>5__4.ReachedNode)

   52             {

   53                 this.<>2__current = this.<wtn>5__4;

   54                 this.<>1__state = 4;

   55                 return true;

   56             Label_0159:

   57                 this.<>1__state = -1;

   58             }

   59             this.<h>5__5 = new Harvesting();

   60             while (!this.<h>5__5.DoneHarvesting)

   61             {

   62                 this.<>2__current = this.<h>5__5;

   63                 this.<>1__state = 5;

   64                 return true;

   65             Label_0198:

   66                 this.<>1__state = -1;

   67             }

   68         }

   69     }

   70     goto Label_01CB;

   71 }

Reflected MoveNext() method

New Year, New Autocrat

For those of you who don't already know, most of Macrocrafter.com's software is built on top of an automation framework called Autocrat. This software allows us to focus our efforts on the task our programs are meant to fulfill, rather than having to deal with the low-level automation-oriented code. As the designer and developer of the Autocrat framework, it's my job to ensure the framework is as easy to use as possible.

Well, in the most current incarnation, Autocrat 3.5, I tried something new. I centered almost all of the functionality around the Window object (which encapsulates an HWND and exposes some window-specific properties and methods). In order to prevent the Window object from becoming a god-object (that is, an object with all the functionality in the entire framework - can you imagine?), I used extension methods to 'add' opt-in functionality. Simply add a using directive for the class of methods you want, and you're good to go.

The problem with being extension-method happy is that extension methods are not incredibly discoverable. It's difficult to know where to look when there are 50-million namespaces hiding as many extension classes. Most of my folly of course came from inexperience with extension methods - they were new, and I wanted to try them out. Alas, it turns out that in several real-life use-cases, extension methods are more harm than good, and don't do much to take away from the so-called god-classes.

To that end, I've decided to remake Autocrat as version 4, to facilitate the creation of EverHarvest 2. About 90% of the work is done, actually - including a new state machine model, a brand new Pointer system, and the beginnings of a new pathwalking framework - so starting on EverHarvest will be a cinch. Not to mention somewhat vital - EverHarvest 1 provided much of the inspiration behind Autocrat 2 and then 3.5, so it seems fitting that EverHarvest 2 facilitate Autocrat 4.

Enough babbling. I'm sure you're thoroughly bored by now - so I'll end with this: my new-year's resolutions.

First, get EverHarvest 2 out the door in beta before Summer rolls around. Second, learn F# finally, by completing at least one of the ideas I blogged about last year. Third, learn the MVC Framework finally, by completing at least one of the ideas I blogged about last year (this actually will be a requirement rather than a resolution given one of my goals at my real job is converting the public-facing website to MVC, probably starting second quarter). Forth, get a hunter bot beta out the door before Summer as well - this will definitely be an integration with EverHarvest. In fact I'm building EH2 with this integration in mind from the start.

So, that's about it. Hope I didn't bore you completely to tears... At least I didn't talk about what I've been doing at work for the past quarter - seriously, dealing with shipping APIs is *not* what I thought I'd be doing when I first started working here... Not fun, not fun - but then again, everyone's got to do some dirty work every now and then. I am grateful for the job, that's for damned sure.

Take care all.

Tuples...

Tuples (which I have always pronounced 'tuh-pul' and others pronounce 'too-pull') are interesting data structures in that they combine two or more pieces of otherwise unrelated data of any types. They are most common (and supported) in functional languages, where they are considered first-class language constructs. For example, to create a Tuple in F#, you simply surround the elements you want to tuplize in a set of parenthesis: [ let x = (1, "one") ] - x in this case will have the type [ int * string ].

You may not realize it yet, but tuples have been around in .NET since 2.0 - the KeyValuePair generic class is, essentially, an immutable tuple of any two values. Using it as such, though, is unwieldy in my opinion - its purpose was to facilitate enumerating through dictionaries, and while it works fine in this context, KeyValuePair is ill-suited to more generic purposes.

During my experimentation with F# and other functional languages, I've come to see the Tuple type as a very valuable data structure that I miss very much when working on "real-world" code (real-world in this context meaning what I do for a living). .NET has no Tuple type, and while I'm aware that .NET 4 will have this type, that doesn't help me now. I could use the F# tuple types in my C# code, yes - but I'd have to distribute the F# binaries with my software, and I don't want to do that if I'm not using F#. My only other options are to use someone else's library, or roll my own. I've opted for the latter.

I've implemented four immutable tuple struct types, which hold between two and 5 values. Additionally, I implemented several helper methods and extension methods that make working with the Tuples a little easier in C#. Here's the code for the two-value Tuple struct:

    1 public struct Tuple<T1, T2> {

    2     private readonly T1 _value1; public T1 Value1 { get { return _value1; } }

    3     private readonly T2 _value2; public T2 Value2 { get { return _value2; } }

    4     public Tuple(T1 value1, T2 value2) { _value1 = value1; _value2 = value2; }

    5 

    6     public override bool Equals(object obj) {

    7         if (!(obj is Tuple<T1, T2>)) return false;

    8         if (obj == null) return false;

    9 

   10         Tuple<T1, T2> t = (Tuple<T1, T2>)obj;

   11         return (Value1.Equals(t.Value1) && Value2.Equals(t.Value2));

   12     }

   13 

   14     public override int GetHashCode() {

   15         return Value1.GetHashCode() ^ Value2.GetHashCode();

   16     }

   17 

   18     public KeyValuePair<T1, T2> AsKeyValuePair() {

   19         return new KeyValuePair<T1, T2>(Value1, Value2);

   20     }

   21 }

   22 ...

   23 

   24 public static class Tuples {

   25     public static Tuple<T1, T2> Tuple<T1, T2>(T1 value1, T2 value2) {

   26         return new Tuple<T1, T2>(value1, value2);

   27     }

   28     ...

   29 

   30     public static Tuple<T1, T2> Default<T1, T2>() {

   31         return new Tuple<T1, T2>(default(T1), default(T2));

   32     }

   33     ...

   34 

   35     public static IEnumerable<Tuple<T1, T2>> Zip<T1, T2>(IEnumerable<T1> first, IEnumerable<T2> second) {

   36         var enum1 = first.GetEnumerator();

   37         var enum2 = second.GetEnumerator();

   38 

   39         while (enum1.MoveNext() && enum2.MoveNext()) {

   40             yield return Tuple(enum1.Current, enum2.Current);

   41         }

   42     }

   43     ...

   44 }

   45 

   46 public static class TupleExtensions {

   47     public static Tuple<IEnumerable<T1>, IEnumerable<T2>> Unzip<T1, T2>(this IEnumerable<Tuple<T1, T2>> ienum) {

   48         var first = new List<T1>();

   49         var second = new List<T2>();

   50 

   51         foreach (var t in ienum) {

   52             first.Add(t.Value1);

   53             second.Add(t.Value2);

   54         }

   55 

   56         return Tuples.Tuple(first.AsEnumerable(), second.AsEnumerable());

   57     }

   58     ...

   59 

   60     public static Tuple<T1, T2> AsTuple<T1, T2>(this KeyValuePair<T1, T2> kvp) {

   61         return Tuples.Tuple(kvp.Key, kvp.Value);

   62     }

   63 }

The '...'s denote where the pattern is extended to cover all the tuple value counts from 2 to 5.

Only a little explanation is really needed here - the Tuple structs are read-only, so once they're initialized they can't be reset. In my experience this isn't a problem - I've never really *needed* that functionality where I can't just create a new Tuple. The static 'Tuples' class makes it easier to initialize a Tuple - using this, I can create a Tuple from existing data without having to add the type parameters. The compiler figures it out from the existing type data. Can't do this with constructors, sadly. The 'AsKeyValuePair' and 'AsTuple' methods (which only work with the 2-tuple struct) are pretty self explanatory.

The Zip method takes two or more IEnumerables and 'zips' them together into a single IEnumerable of Tuples. The Unzip method sort performs the reverse, although since you can only return one value from a method I package the unzipped Enumerables in a single Tuple.

If you'd like to use this in your own projects, I've uploaded my Tuples file to PasteBin - you can get to it from here. No attribution needed - though it would be nice if you'd drop a line here to let me know it's been useful to you. =)