Jacob Reimers' Blog

Reactive Extensions is More Than Async

Sun, 08 Apr 2012 15:17:44 GMT

A short while ago I asked for some feedback for the latest addition to the Linq2Rest project, Linq2Rest.Reactive, and the feedback I got started some thinking about why I was going for one solution over another. Peter Goodman asked a good question, when he asked why I did not merely wrap the LINQ query in a Task<T>. My initial thought was that the reactive addition to Linq2Rest would be an asynchronous version of querying REST services, but when you think about it, then Rx is more than merely an asynchronous version of LINQ.

A lot is being said and written about asynchronous programming, and with the introduction of the new async/await keywords in C# 5 everyone will soon be doing it. Linq2Rest.Reactive is also about asynchronous querying, but if that was the only problem to solve, I would take Pete's suggestion and create a wrapper for a LINQ query. I did that in the base Linq2Rest package with the ExecuteAsync extension method to an IQueryable<T>.

Asynchronous programming is hard. Async/await will make writing asynchronous code easier, but there are still many things to consider about threading, execution paths and availability. If you don't understand asynchronicity and what deferred execution is, then async/await won't help you anymore than a calculator will help you if you don't understand math. Yes, you can get a result. You may even get a reasonably good result, but if you don't understand why, then you will have a hard time using it in combination with other things.

Reactive Extensions (Rx) offers more than just a way to get results asynchronously. With Rx you get the ability to compose and merge queries. This allows for richer queries as well as a more declarative way to build up the data to be consumed. On top of this you get the option to decide which thread context you wish to handle your data on. Instead of working this out for yourself, you can leverage the brains of the Rx team when it comes to synchronizing your data.

One of the primary use cases for Rx is working with streams of data. As Peter rightly pointed out, reactive programming implies observing something more than a single result set, because this is effectively what an asynchronous query execution is. Since REST is based on a polling model, it is by the nature of things not compatible with streams. It is much more like awaiting a single burst of results when queried. Setting up an observable (which is presumed to be longer lived) may seem a bit excessive. If on the other hand the single result is your only concern, then it does seem more suitable to use asynchronous execution.

However if the data you are working with is continuous, then it may make sense to set up an observable context, even though the data has to be fetched using polling. This could be the case with an RSS feed, which is assumed to be updated continually. There are lots of other examples of data, such as financial data, which changes with regular intervals, which can be polled. With the advent of web sockets continuous connections will become more frequent, and so will web based data streams. Linq2Rest is designed to not care whether you are using polling or long-lived connections.

Going back to the original point of the post, then async querying may work well for many scenarios, and does seem the logical default choice when it comes to REST based data. But from a querying perspective, reactive extensions provide an alternate approach to working with the data, which may become more popular with the growth of long lived connections. Whichever you chose will depend on the data you are working with and how you intend to use it. For me it is important that Linq2Rest be able to support either type of usage. The important thing is that you understand why you are choosing one over the other.

Queryable Web Services - Parsing Query Strings to LINQ

Mon, 19 Mar 2012 09:15:40 GMT

I have previously made a stab at supposedly RESTful webservices where the response type is determined by a part of the request URI and not the Accept header as it should be. This time I have been contemplating another problem with the current approach to RESTful web services, which is the difficulty in querying them.

We all know the standard RESTful URI like:

http://domain/blogs/name/123

This will give the blog post with the specified id.

The above URI structure is also seen as:

http://domain/blogs/name

This is expected to give all posts for that particular blog.

But what if I wanted to have blog post that mention “foo”? To my knowledge there is no way to specify that in a standard way. The individual web service can allow you to specify custom parameters which may help you in filtering posts. The other approach is to download all the posts and then filtering them locally. The latter approach is of course hugely wasteful and is better done on the server.

As I mentioned, the individual web service can specify some custom capabilities, but I think a more standardized approach is needed.

The OData protocol specifies some query string options to perform filtering, sorting and projection. Much like LINQ operators, but as query parameters. From above, an OData service will be able to perform filtering so that I can get only posts containing “foo” by specifying that in a defined parameter, like so:

http://domain/blogs/name?$filter=substringof(‘foo’, Text)&$orderby=Date

Obviously one would have to learn the syntax, but there is huge potential in having a known query syntax. There are a number of OData server and client libraries. Unfortunately for .NET I couldn’t find anything that would help me use it in a custom web service or an existing ASP.NET or ASP.NET MVC site, so I wrote up a parser that reads an HTTP request and creates a LINQ query from the query strings defined in the OData specification. You can find the code on BitBucket. ~~It's an initial drop, so be gentle :-)~~

Currently it supports a subset of the specification, but if you have a look at the tests, you will see that it can parse requests like:

http://domain/blah?$filter=substring(StringValue, 1) ne 'text' and IntValue eq 25 and DoubleValue le 10

into the following lambda expression:

x => (((x.StringValue.Substring(1) != "text") && (x.IntValue == 25)) && (x.DoubleValue <= 10))

The starting point is the ParameterParser class which has a Parse method which goes through the query parameters and returns a ModelFilter which holds the LINQ query generated from the query parameters. The ModelFilter class has a Filter method which applies the LINQ query to a collection of model items. This could be the blog posts from the example above. Although I haven't tested it yet, it should also be compatible with Entity Framework or NHibernate so that it is possible to apply this filter to the DbContext (Entity Framework) or the Query (Nhibernate) to generate a filtered database request. The whole point of parsing into a LINQ query is that you now have the benefit of being able to filter on the server in a consistent way.

The project includes a ModelFilterBinder class as well, so that it is possible to have the ModelFilter as a parameter to controller actions in ASP.NET MVC. For classical ASP.NET sites, the ParameterParser will have to do.

I am aware that creating a blanket implementation of this filtering may not suit all scenarios. For example a web site that only supports HTML responses may not want to support it. On the other hand a, on a blog site it may well make sense to support filtering and adapt the rendering accordingly. For web services that want to present themselves as RESTful I see little downside in supporting this. Remember that there is always the option of returning a 400 Bad Request response.

The UrlQueryParser doesn't support projection yet, and this will certainly have a bigger impact on HTML responses as certain properties from the model may not be available for rendering. For example if I project a blog into only Author and Date with no Text. Again a 400 Bad Request response is an option, another option is coming up with a flexible layout that does support this.

Creating DTOs at Runtime

Sun, 05 Feb 2012 23:30:42 GMT

One of the first problems that came up when creating the Linq2Rest project was how to handle LINQ projections (select statements). On the client the resulting anonymous type would be generated because it is part of the compiled code, but the server that has to deserialize and execute the LINQ query has no knowledge of the projected type. Still it has to be able to handle the request.

One way to handle this would obviously be to use the resource type in full and only set the requested values. This becomes very visible during serialization and may cause problem when deserializing the response on the client. Another unsuitable solution is to generate types for all projections, but if your initial type is large, this becomes pointless. A nicer way is to generate a matching DTO (data transfer object) at runtime which can be serialized and returned to the client. It should be simple enough, basically it only needs an automatic property so it can store the value before serialization. It turns out that creating a runtime type is surprisingly easy, and so are automatic properties (once you know how).

The easiest solution is to create a type with public fields. This takes care of most scenarios. But let's go for the full property.

The first step is to create a ModuleBuilder in order to get access to the builder classes:

var moduleBuilder = Thread
	.GetDomain()
	.DefineDynamicAssembly(AssemblyName, AssemblyBuilderAccess.Run)
	.DefineDynamicModule(AssemblyName.Name);

This gives you an instance of a ModuleBuilder. With this you can create your runtime DTO type, using a TypeBuilder.

var typeBuilder = ModuleBuilder.DefineType(className, TypeAttributes.Public | TypeAttributes.Class | TypeAttributes.Serializable);

The exact type attributes will be up to you, but you'll probably want to create a public class. Next it is time to define a backing field, a property and getter and setter methods.

var fieldBuilder = typeBuilder.DefineField(fieldName, propertyType, FieldAttributes.Private);

var propertyBuilder = typeBuilder.DefineProperty(propertyName, PropertyAttributes.None, propertyType, null);

var getAccessor = typeBuilder.DefineMethod(
	"get_" + propertyName,
	MethodAttributes.Final | MethodAttributes.Public,
	propertyType,
	Type.EmptyTypes);

var getIl = getAccessor.GetILGenerator();
getIl.Emit(OpCodes.Ldarg_0);
getIl.Emit(OpCodes.Ldfld, fieldBuilder);
getIl.Emit(OpCodes.Ret);

var setAccessor = typeBuilder.DefineMethod(
	"set_" + propertyName,
	MethodAttributes.Final | MethodAttributes.Public
	null,
	new[] { propertyType });

var setIl = setAccessor.GetILGenerator();
setIl.Emit(OpCodes.Ldarg_0);
setIl.Emit(OpCodes.Ldarg_1);
setIl.Emit(OpCodes.Stfld, fieldBuilder);
setIl.Emit(OpCodes.Ret);

propertyBuilder.SetGetMethod(getAccessor);
propertyBuilder.SetSetMethod(setAccessor);

That's it. The code should be fairly self-explanatory. First the backing field is created using the DefineField method. Next the property is created with the DefineProperty method. The biggest gotcha is that you have to create getter and setter methods (after all, it's not Java). Defining the method is pretty much the reverse process of what you would do to find them using reflection. The tricky bit is creating the method body. But once you have the recipe for how to create the IL that goes into these methods, they are easy to make. Finally the getter and setter methods are attached to the property.

Once the necessary properties have been created, the Type needs to be loaded, which is done by having the TypeBuilder create the type, like so:

typeBuilder.CreateType();

Once the type is created it would make sense to store it somewhere where you can retrieve it next time someone requests a matching type.

Linq2Rest will create an instance of the runtime DTO and populate the properties before it is returned to the user. This is where it is useful to have a DTO that uses properties, because it will look like the requested anonymous type from a reflection point of view. This may not be important if the DTO is immediately serialized and sent to the user. It may be useful is the object is returned to the user in some other way where the type is preserved.

Another question is how to handle types that have defined serialization aliases. This causes a mismatch between the publicly exposed property names and the ones actually used to execute the LINQ query. That will be for another post.

MVVM. de la découverte à la maîtrise - A Book Review

Tue, 27 Dec 2011 10:37:11 GMT

I just finished reading "MVVM. de la découverte à la maîtrise" (MVVM refers to the Model View ViewModel pattern if you need to ask). To my knowledge it is one of the only books that covers the full spectrum of topics related to developing an application using the MVVM pattern. In short , this is the book that teaches you how to structure and develop your WPF, Silverlight and WindowsPhone applications.

The book does however have a huge drawback - it's in French! Hopefully it will be translated soon, because it deserves a wider audience than just the French.

In English, the title of the book translates to something like: MVVM, From Discovering to Mastering. This gives you a hint that the target audience is people who are beginners when it comes to MVVM and want to develop their skills. The book does pretty much that. It is a guide through the different parts that make up the MVVM pattern with code samples to show you how the advice would be implemented in practice.

Even though I consider myself somewhat beyond the beginner level when it comes to MVVM (IMHO) I wanted to read this book because, as I mentioned above, there are so few books that deal with the full MVVM pattern. If you are content to read through a ton of blog posts about MVVM or ViewModels in particular, then the web is full of them. I do however like to read a comprehensive approach to a subject, and blog posts are useless for this.

The book is a well structured walk-though of the different components starting from an introduction of the general topic area to the different parts of MVVM, i.e. a chapter for describing how to handle the Model, another to describe the ViewModel and finally a long chapter about the View. The text takes a pragmatic approach to the code and focuses on solving the problem rather than on insisting on purity, which makes it a good practical guide for the novice in the world of MVVM. Since the book does target developers who are new to MVVM, it places much more emphasis on how to do things than on why things should be done in a particular way. The model gets less attention than the ViewModel and the View which is normal since that is more of a solved problem than the ViewModel and View whose implementation is specific to MVVM. Both the chapter about the ViewModel and the View are supported by a sample library application with code snippets to examplify a particular section. I'll get into some of the examples below, but for a beginner I would say that it is generally easy to approach while still having some expert features to bring the reader beyond the sandbox stage. As mentioned above, there is still some room to add advice to the master by adding more descriptions about the things work under the hood and how this influences choices when it comes to choosing an implementation.

The book ends with a chapter about testing you application, both as manual testing, unit testing and UI automation testing. For me this was the most disappointing chapter to read. It had been mentioned throughout the book. However when I finally got to it, it was very superficial and dealt mainly with ViewModel testing in WPF. I am fully aware of the limitations when it comes to testing Silverlight and WindowsPhone. The authors can't be critized for the complete lack of attention from Microsoft in this area, but I had hoped that the book would have contained some advice on how to work around the limitations.

I don't like nitpicking, especially not when it comes to such a large effort as this, but there are things in this book that I was surprised to see had made it to print (or in my case, to display on my Kindle).

The most important issue that I had with the book was the general lack of clear definitions when it comes to core concepts. For example, the ViewModel is described at one point as an abstraction of the View. While there is some truth in this, it clashes with the constant emphasis that the book places on decoupling the different parts of the MVVM pattern. To be very pedantic, then if the ViewModel is an abstraction of the View it must have a relation with the View (otherwise the relationship is coincidental). But if the ViewModel has a relationship with the View a coupling from the View to the ViewModel has been created. To me the ViewModel must not be coupled to the View at all and can therefore not conceptually be an abstraction of the View. Instead I would have preferred the ViewModel to be defined as a Viewable Model which depends on the Model. This way the dependency links are the right way round and the ViewModel does not need to know about the View. The diagrams in the book do actually show this but the text contradicts the diagram. What's worse is when a ViewModel is compared to apple pie. I think you have to be French to understand that comparison.

In extension of this the pragmatic approach that the authors have taken to problem solving leads to examples where View concepts bleed into the ViewModel. They do make a point to emphasize the need for distinction, but I was still left with a doubt as to how clear they wanted to make the separation. My own preference is to keep anything that smells of graphics out of the ViewModel, but sometimes the code samples blur the lines. This is probably acceptable for entry level applications, but can cause problems when it comes to enterprise level applications.

The next thing is that the code samples have erased all type names. This leads to some very odd code samples. For example, it is left up to the reader to guess that .NewGuid() is actually a call to Guid.NewGui(). That is reasonably easy to guess, but it becomes more difficult when it comes to generic type parameters. I'm not quite sure how something like that escapes proof reading, and it may only be present on the Kindle, but it is definitely annoying. In other places you stumble across code like this:

string.Format("Some text{0}{1}Some names", Environment.NewLine, Environment.NewLine);

Finally there are some odd jumps in the level of detail that is used in the examples. At one point the book describes the benefits of having strongly typed ViewModels and how this can be done using generics. Then some time is spent explaining generics, whereas when it comes to delegrates it is assumed that the reader is fully acquainted with them. I mention the example with generics specifically because it is the one thing the View doesn't care about, since the View binds itself to properties using reflection, and so mainly has value when it comes to maintaining the ViewModel.

In conclusion I would say that I wrote this review as a recommendation to go and read the book. Even if you think that you know everything about MVVM the book makes you think about your preconceptions. I hope it gets translated to English soon so it can be read by more people.

Accept Headers for HTML Tags

Thu, 28 Jul 2011 19:59:40 GMT

RESTful services are in fashion at the moment, but there is some disagreement as to what actually constitutes RESTfulness (if that is an actual word). A lot of times I hear that a service is RESTful because it uses HTTP or because it has a pretty URL. One thing I rarely hear is an emphasis on different representations of a given resource.

Consider the the two different URLs:

http://domain/resource.xml
http://domain/resource.json
http://domain/xml/resource
http://domain/json/resource

They are common ways of expressing different representations of a resource, which is supposedly the same. The human reader can may read them as the same, but they are different URLs by the definition of the HTTP standard. They should all really just be http://domain/resource.

One of my main annoyances when looking at supposedly RESTful services cannot return different representations. When the user sends a request it should be possible to specify in which form he wants the response. According to the HTTP standard, this is done, as you most likely know, by setting the Accept header. There may be limits to the capabilities of the service, but it should not be necessary to request different resources (by which I mean different URLs).

This is fine and well on the service end, but the consumer should also play by the rules and send a request with an Accept header that gives the service a chance to respond correctly. Arguably the browser is the primary REST consumer, so it should also be a requirement that it generates requests with proper Accept headers, not only for a page itself, but for the content that is being requested in the individual content elements, most notably the script and image elements.

I set up a resource that shows the location of the requesting IP address at IP Location. The basic HTML representation is:

<script src="/ip" type="text/javascript"></script>
<h2>IpLocation</h2>
<p>Location: 12,34 </p>
<div>
	<img alt=", " src="/ip" />
</div>

The idea is that when the browser requests the URL it should generate three requests, each for their own representation. The initial request is expected to request an HTML representation. The image tag on the page is expected to request an image representation and the script tag is expected to request a JavaScript representation. If all goes well the page should be showing text, an image and have a script element with a JSON object.

If you haven't already clicked the link, I can reveal right now, that Internet Explorer is the only one that sends the correct headers and displays the content in the browser as intended.

Looking at the headers from the different tested browsers we can see that Chrome starts with a proper header for the initial request, which means it gets a sensible HTML response. But for the image and the script elements, the Accept headers are simply */*. This basically means that Chrome is saying: "Give it to me any way you want". That is probably the safest way of getting a useless response, and it seems strange that Chrome doesn't even attempt to send something that is even remotely relevant.

Chrome Accept header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept: */*
Accept: */*

Firefox is a bit better, sending correct Accept headers in two out three requests. Only for the script request does it send a */* blanket request.

Firefox 5 Accept header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept: */*
Accept: image/png,image/*;q=0.8,*/*;q=0.5

As revealed above, Internet Explorer 9 does it right three out of three times.

IE 9 Accept header:
Accept: text/html, application/xhtml+xml, */*
Accept: application/javascript, */*;q=0.8
Accept: image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5

So if you do are going to view the location in a browser, I would suggest doing it in IE9.

UI Interaction Using the Kinect and WPF

Sat, 18 Jun 2011 16:57:33 GMT

I pushed an initial drop of code to hook up the Kinect to a WPF application using the Kinect SDK. The code is committed to BitBucket in the Kinect.WPF project. The main goal is to have a way to trigger UI events from body movements. The SDK doesn't install any kind of input device so there is no "official" way to translate the user movements into UI interaction, such as with a mouse device or a stylus.

The main structure of the project is to hook the KinectConnection to the window as an attached property. The KinectConnection takes in a Sensor, which is a wrapper around the Kinect device which tracks the skeleton frame. It also takes a number of KinectCursors, which are in effect a specific joint mapped as a cursor moving over the UI and triggering events. You can wire it up like so:

XAML

<Window x:Class="Kinect.App.MainWindow"
		xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
		xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
		xmlns:Tools="clr-namespace:Kinect.Tools;assembly=Kinect.Tools"
		Title="MainWindow"
		Height="600"
		Width="800">
	<Tools:KinectConnection.Connection>
		<Tools:KinectConnection>
			<Tools:KinectConnection.Sensor>
				<Tools:Sensor />
			</Tools:KinectConnection.Sensor>
			<Tools:KinectConnection.Cursors>
				<Tools:KinectCursor x:Name="rightCursor"
									Joint="HandRight" />
				<Tools:KinectCursor x:Name="leftCursor"
									Joint="HandLeft" />
			</Tools:KinectConnection.Cursors>
		</Tools:KinectConnection>
	</Tools:KinectConnection.Connection>
</Window>

As you can see it is hooked up as a normal attached property. The sensor is a simple instance, and the cursors are linked to a joint by setting the Joint property. Internally the KinectConnection attempts to determine the current window as work from there. The movements are defined relative to the window.

WPF makes it reasonably easy to create your own UI events by declaring RoutedEvents. Currently the RoutedEvents declared by the KinectCursor include CursorEnter and CursorLeave, as well as CursorUp and CursorDown. These are just the initial events, and I hope to be adding more in the future. More important are the combinations of cursor movements into gestures, such as when hands move together in a certain way, such as for example a pinch movement with the hands.

In the sample app in the Kinect.WPF project the UI is set to react to the KinectCursor entering and leaving. In the project this is done by setting an EventTrigger in the control style, but a handler could be attached in other standard ways as well. An example for the textblock is:

XAML

<Style x:Key="textStyle"
		TargetType="TextBlock">
	<Style.Triggers>
		<EventTrigger RoutedEvent="Tools:KinectCursor.CursorEnter">
			<BeginStoryboard>
				<Storyboard>
					<DoubleAnimation To="28"
										Duration="0:0:0.5"
										AccelerationRatio="0.10"
										DecelerationRatio="0.25"
										Storyboard.TargetProperty="(TextElement.FontSize)" />
				</Storyboard>
			</BeginStoryboard>
		</EventTrigger>
	</Style.Triggers>
</Style>

As you can see, on the CursorEnter event the font size is changed.

This is very close to the current way that the mouse interacts currently. The main difference is obviously that there can be multiple KinectCursors. The cursors also need to expose more detailed motion data. This is what has to be utilized to define motion gestures, which is the bigger goal of the project. Follow the project for updates on this.

I'm just getting started on the project, and if you want to fork the code, feel free. I should also give due credit to the Kinect Paint project, which has inspired some of the code.

Added Google Analytics Reader for .NET

Tue, 14 Jun 2011 19:08:50 GMT

I found out yesterday that Google recently launched anAnalytics API. That is a very welcome to their APIs. But I was surprised to find that .NET developers were left to create their own HTTP requests and parse the results. I'm not going to go into any conspiracy theories here, but they managed to support Java, so why not .NET?

Well if Google won't support .NET, then I will :-) I added Analytics support to the Reimers.Google project in the public SVN trunk (svn://svn.reimers.dk/Reimers.Google). There will also be some code later in this post if you don't know how to use SVN.

The workhorse is the ReportRequestor class which handles the login and report requests and parsing. To read an Analytics report you have to create an instance of the ReportRequestor and set the username and password.

Before you can generate a report you will need to find the Analytics account you want to request the report for. Get all your available Analytics account by calling the GetAccounts method and selecting the one you want to use, like so:

IEnumerable accounts = requestor.GetAccounts();
AnalyticsAccountInfo account = accounts.First(a => a.Title == "Reimers.dk");

Once you have your account information you can use it to request a report. The ReportRequestor exposes the RequestReport method which returns an IEnumerable. The GenericEntry is basically just a container for the Dimensions and Metrics you requested.

In the project code there is also a GetUserCountByLocation method which shows how you can take the GenericEntry list and generate a strongly typed entry based on your own business logic.

All the response parsing is done using LINQ, so you will need .NET 3.5. But then you should be ready to generate your reports.

The complete source code is below. You can use it for your own personal or commercial projects, but please remember to include credits. If you find any bugs or want assistance in generating you Analytics reports send me a message.

ReportRequestor

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Xml.Linq;
using Reimers.Google.Analytics.Reports;
using System.Globalization;

namespace Reimers.Google.Analytics
{
	public class ReportRequestor
	{
		#region Fields

		private static readonly string requestUrlFormat = "https://www.google.com/analytics/feeds/data?ids={0}&dimensions={1}&metrics={2}&start-date={3}&end-date={4}";
		private static readonly string authUrlFormat = "accountType=GOOGLE&Email={0}&Passwd={1}&source=reimers.dk-analyticsreader-0.1&service=analytics";
		private static CultureInfo ci = CultureInfo.GetCultureInfo("en-US");
		private string _token = null;
		private string _username = null;
		private string _password = null;

		#endregion

		#region Constructor

		public ReportRequestor() { }

		public ReportRequestor(string email, string password)
		{
			_username = email;
			_password = password;
		}

		#endregion

		#region Properties

		public string Email
		{
			get { return _username; }

			set
			{
				if (!string.Equals(_username, value))
				{
					_username = value;
					_token = null;
				}
			}
		}

		public string Password
		{
			get { return _password; }
			set
			{
				if (!string.Equals(_password, value))
				{
					_password = value;
					_token = null;
				}
			}
		}

		#endregion

		#region Methods

		private string GetToken(string username, string password)
		{
			if (string.IsNullOrEmpty(_username) || string.IsNullOrEmpty(_password))
			{
				throw new ArgumentNullException("Username, Password", "Username and/or password not set");
			}

			string authBody = string.Format(authUrlFormat, username, password);
			HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("https://www.google.com/accounts/ClientLogin");
			req.Method = "POST";
			req.ContentType = "application/x-www-form-urlencoded";
			req.UserAgent = "Reimers.dk req";

			Stream stream = req.GetRequestStream();
			StreamWriter sw = new StreamWriter(stream);
			sw.Write(authBody);
			sw.Close();
			sw.Dispose();

			HttpWebResponse response = (HttpWebResponse)req.GetResponse();
			StreamReader sr = new StreamReader(response.GetResponseStream());
			string token = sr.ReadToEnd();
			string[] tokens = token.Split(new string[] { "\n" }, StringSplitOptions.RemoveEmptyEntries);

			foreach (string item in tokens)
			{
				if (item.StartsWith("Auth="))
				{
					return item.Replace("Auth=", "");
				}
			}

			return string.Empty;
		}

		public IEnumerable GetAccounts()
		{
			if (string.IsNullOrEmpty(_token))
			{
				_token = GetToken(_username, _password);
			}

			HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("https://www.google.com/analytics/feeds/accounts/default");
			req.Headers.Add("Authorization: GoogleLogin auth=" + _token);
			HttpWebResponse response = (HttpWebResponse)req.GetResponse();

			Stream responseStream = response.GetResponseStream();
			StreamReader sr = new StreamReader(responseStream);
			string responseXml = sr.ReadToEnd();

			XDocument doc = XDocument.Parse(responseXml);
			XNamespace dxpSpace = doc.Root.GetNamespaceOfPrefix("dxp");
			XNamespace defaultSpace = doc.Root.GetDefaultNamespace();

			var entries = from en in doc.Root.Descendants(defaultSpace + "entry")
						  select new AnalyticsAccountInfo
						  {
							  AccountID = en.Elements(dxpSpace + "property").Where(xe => xe.Attribute("name").Value == "ga:accountId").First().Attribute("value").Value,
							  AccountName = en.Elements(dxpSpace + "property").Where(xe => xe.Attribute("name").Value == "ga:accountName").First().Attribute("value").Value,
							  ID = en.Element(defaultSpace + "id").Value,
							  Title = en.Element(defaultSpace + "title").Value,
							  ProfileID = en.Elements(dxpSpace + "property").Where(xe => xe.Attribute("name").Value == "ga:profileId").First().Attribute("value").Value,
							  WebPropertyID = en.Elements(dxpSpace + "property").Where(xe => xe.Attribute("name").Value == "ga:webPropertyId").First().Attribute("value").Value
						  };

			return entries;
		}

		private XDocument getReport(AnalyticsAccountInfo account, IEnumerable dimensions, IEnumerable metrics, DateTime from, DateTime to)
		{
			if (string.IsNullOrEmpty(_token))
			{
				_token = GetToken(_username, _password);
			}

			StringBuilder dims = new StringBuilder();

			foreach (Dimension item in dimensions)
			{
				dims.Append("ga:" + item.ToString() + ",");
			}

			StringBuilder mets = new StringBuilder();

			foreach (Metric item in metrics)
			{
				mets.Append("ga:" + item.ToString() + ",");
			}

			string requestUrl = string.Format(requestUrlFormat, "ga:" + account.ProfileID, dims.ToString().Trim(",".ToCharArray()), mets.ToString().Trim(",".ToCharArray()), from.ToString("yyyy-MM-dd"), to.ToString("yyyy-MM-dd"));

			HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(requestUrl);
			req.Headers.Add("Authorization: GoogleLogin auth=" + _token);
			HttpWebResponse response = (HttpWebResponse)req.GetResponse();

			Stream responseStream = response.GetResponseStream();
			string responseXml = new StreamReader(responseStream, Encoding.UTF8, true).ReadToEnd();
			XDocument doc = XDocument.Parse(responseXml);

			return doc;
		}

		public IEnumerable RequestReport(AnalyticsAccountInfo account, IEnumerable dimensions, IEnumerable metrics, DateTime from, DateTime to)
		{
			XDocument doc = getReport(account, dimensions, metrics, from, to);
			XNamespace dxpSpace = doc.Root.GetNamespaceOfPrefix("dxp");
			XNamespace defaultSpace = doc.Root.GetDefaultNamespace();

			var gr = from r in doc.Root.Descendants(defaultSpace + "entry")
					 select new GenericEntry
					 {
						 Dimensions = new List>(
							 from rd in r.Elements(dxpSpace + "dimension")
							 select new KeyValuePair(
																 (Dimension)Enum.Parse(
																 typeof(Dimension),
																 rd.Attribute("name").Value.Replace("ga:", ""),
																 true),
																 rd.Attribute("value").Value)),
						 Metrics = new List>(
							 from rm in r.Elements(dxpSpace + "metric")
							 select new KeyValuePair(
								 (Metric)Enum.Parse(typeof(Metric), rm.Attribute("name").Value.Replace("ga:", ""), true),
								 rm.Attribute("value").Value))
					 };

			return gr;
		}

		public IEnumerable GetUserCountByLocation(AnalyticsAccountInfo account, DateTime from, DateTime to)
		{
			IEnumerable report = RequestReport(account, new Dimension[] { Dimension.city, Dimension.latitude, Dimension.longitude }, new Metric[] { Metric.visits }, from, to);

			var cr = from r in report
					 select new CityReport
										 {
											 City = r.Dimensions.First(d => d.Key == Dimension.city).Value,
											 Latitude = Convert.ToDouble(r.Dimensions.First(d => d.Key == Dimension.latitude).Value, ci),
											 Longitude = Convert.ToDouble(r.Dimensions.First(d => d.Key == Dimension.longitude).Value, ci),
											 Count = Convert.ToInt32(r.Metrics.First(m => m.Key == Metric.visits).Value)
										 };

			return cr;
		}

		#endregion
	}
}

AnalyticsAccountInfo

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Reimers.Google.Analytics
{
	public class AnalyticsAccountInfo
	{
		#region Fields

		private string _id = null;
		private string _title = null;
		private string _accountId = null;
		private string _accountName = null;
		private string _profileId = null;
		private string _webProperty = null;

		#endregion

		#region Properties

		public string ID
		{
			get { return _id; }
			set { _id = value; }
		}

		public string Title
		{
			get { return _title; }
			set { _title = value; }
		}

		public string AccountID
		{
			get { return _accountId; }
			set { _accountId = value; }
		}

		public string AccountName
		{
			get { return _accountName; }
			set { _accountName = value; }
		}

		public string ProfileID
		{
			get { return _profileId; }
			set { _profileId = value; }
		}

		public string WebPropertyID
		{
			get { return _webProperty; }
			set { _webProperty = value; }
		}
		#endregion
	}
}

GenericEntry

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Reimers.Google.Analytics
{
	public class GenericEntry
	{
		#region Fields

		List> _dimensions = null;
		List> _metrics = null;

		#endregion

		#region Properties

		public List> Dimensions
		{
			get { return _dimensions; }
			set { _dimensions = value; }
		}

		public List> Metrics
		{
			get { return _metrics; }
			set { _metrics = value; }
		}

		#endregion
	}
}

CityReport

namespace Reimers.Google.Analytics.Reports
{
	public class CityReport
	{
		#region Fields

		private string _city = string.Empty;
		private double _lat = 0;
		private double _lng = 0;
		private int _count = 0;

		#endregion

		#region Properties

		public string City
		{
			get { return _city; }
			set { _city = value; }
		}

		public double Latitude
		{
			get { return _lat; }
			set { _lat = value; }
		}

		public double Longitude
		{
			get { return _lng; }
			set { _lng = value; }
		}

		public int Count
		{
			get { return _count; }
			set { _count = value; }
		}

		#endregion
	}
}

Dimension

namespace Reimers.Google.Analytics
{
	public enum Dimension
	{
		browser = 0,
		browserVersion = 1,
		city = 2,
		connectionSpeed = 3,
		continent = 4,
		countOfVisits = 5,
		country = 6,
		date = 7,
		day = 8,
		daysSinceLastVisit = 9,
		flashVersion = 10,
		hostname = 11,
		hour = 12,
		javaEnabled = 13,
		language = 14,
		latitude = 15,
		longitude = 16,
		month = 17,
		networkDomain = 18,
		networkLocation = 19,
		pageDepth = 20,
		operatingSystem = 21,
		operatingSystemVersion = 22,
		region = 23,
		screenColors = 24,
		screenResolution = 25,
		subContinent = 25,
		userDefinedValue = 26,
		visitorType = 26,
		week = 27,
		year = 28,
		adContent = 29,
		adGroup = 30,
		adSlot = 31,
		adSlotPosition = 32,
		campaign = 33,
		keyword = 34,
		medium = 35,
		referralPath = 36,
		source = 37,
		exitPagePath = 38,
		landingPagePath = 39,
		pagePath = 40,
		pageTitle = 41,
		affiliation = 42,
		daysToTransaction = 43,
		productCategory = 44,
		productName = 45,
		productSku = 46,
		transactionId = 47,
		searchCategory = 48,
		searchDestinationPage = 49,
		searchKeyword = 50,
		searchKeywordRefinement = 51,
		searchStartPage = 52,
		searchUsed = 53
	}
}

Metric

namespace Reimers.Google.Analytics
{
	public enum Metric
	{
		bounces = 0,
		entrances = 1,
		exits = 2,
		newVisits = 3,
		pageviews = 4,
		timeOnPage = 5,
		timeOnSite = 6,
		visitors = 7,
		visits = 8,
		adCost = 9,
		adClicks = 10,
		CPC = 11,
		CPM = 12,
		CTR = 13,
		impressions = 14,
		uniquePageviews = 15,
		itemQuantity = 16,
		itemRevenue = 17,
		transactionRevenue = 18,
		transactions = 19,
		transactionShipping = 20,
		transactionTax = 21,
		uniquePurchases = 22,
		searchDepth = 23,
		searchDuration = 24,
		searchExits = 25,
		searchRefinements = 26,
		searchUniques = 27,
		searchVisits = 28,
		goal1Completions = 29,
		goal2Completions = 30,
		goal3Completions = 31,
		goal4Completions = 32,
		goalCompletionsAll = 33,
		goal1Starts = 34,
		goal2Starts = 35,
		goal3Starts = 36,
		goal4Starts = 37,
		goalStartsAll = 38,
		goal1Value = 39,
		goal2Value = 40,
		goal3Value = 41,
		goal4Value = 42,
		goalValueAll = 43
	}
}

Using the OverlayManager with Silverlight Maps

Tue, 14 Jun 2011 19:08:14 GMT

Deciding to switch to Microsoft's Silverlight map control was not without concern. While it is a great platform for displaying maps in Silverlight and gives you the full rich Silverlight UI engine, it does lack some user friendliness. One of the problems with getting started with Silverlight (or WPF) is the overwhelming amount of options you suddenly have. But with all these options come the requirement to manage them. This was one of my first concerns with the Silverlight map control. While it allows you to display rich overlays on the map relatively easy, it doesn't provide you with an easy interface to handle the overlays.

This is why I created the OverlayManager. It hooks on to a map and handles how to display overlays, and order them so your polygon does not cover all your markers, etc. It does this by having it's own layers (MapLayer) so it can easily order overlays by type. To use the OverlayManager, since create it and pass in the map you want it to hook on to, like so:

OverlayManager manager = new OverlayManager(map);

Now you are ready to create your overlays. Here we just create a circle and put it on the map:

Ellipse circle = new Ellipse { Width = 10, Height = 10, Fill = new SolidColorBrush(Colors.Green) };
manager.Add(circle, new Location(51.47,0.0));

As you can see you don't actually add it to the map, but instead add it to the OverlayManager, which in turns figures out how to plot it. You don't have to worry about attached properties and so on.

Another thing I found missing with the map control is the lack of KML support, so I added that to the OverlayManager as well. KML is the current standard for exchanging geographic data, so I was surprised that it was not included. However the OverlayManager makes it easy to read and write compliant KML, simply pass your KML document to the ReadKml method and it will read through it and create the overlays that are compatible with the Silverlight map control.

string kml = "your kml document";
manager.ReadKml(kml);

I will go into more detail about the more advanced features, KML styling, displaying infowindows, etc, of the OverlayManager in later posts.

Ensuring Valid SqlGeography Objects

Tue, 14 Jun 2011 19:00:31 GMT

Yesterday I wrote a post about the release of a project that can convert shapefiles and KML files to SqlGeography objects for use in a SQL Server 2008 spatial database. The big issue was that the SqlGeography class won't create valid objects from any collection of points. First of all the drawn polygon cannot be larger than a hemisphere. You would think that Microsoft's coders would be smart enough to render the input as the smaller of the two options. Google Earth does not require your KML files to be ordered in a special way. The second problem is that polygon border cannot cross themselves. Again you would think that this could have been handled by some input validation, but no.

I received a comment from k.van.daalen who explained how he had managed to clean his database by SQL script. Since the SQL scripts call CLR functions, it is possible to do the same using code. Below you can see the process of generating valid SqlGeography objects. The input is a GooglePolygon which can be created either using the Shapefile reader or by feeding a KML definition to a GoogleOverlayCollection.

private static void WritePolygon(GooglePolygon shape, SqlGeographyBuilder b)
{
	if (shape.Points[0] != shape.Points[shape.Points.Count - 1])
	{
		shape.Points.Add(shape.Points[0].Clone());
	}

	b.BeginGeography(OpenGisGeographyType.Polygon);
	b.BeginFigure(shape.Points[0].Latitude, shape.Points[0].Longitude);

	for (int i = 1; i < shape.Points.Count; i++)
	{
		b.AddLine(shape.Points[i].Latitude, shape.Points[i].Longitude);
	}

	b.EndFigure();
	b.EndGeography();
}

When the polygon has been written to the SqlGeographyBuilder the SqlGeography object can be retrieved from the ConstructedGeography. If the written polygon is not valid an exception is thrown. The builder doesn't have any way to know if the built object is valid, so you are left with catching the exception.

The catch code goes like this:

As k.van.daalen told, the SqlGeometry class has a MakeValid method which is not present in the SqlGeography class (why??). So the polygon is recreated as a SqlGeometry object which is then 'made valid'.

catch
{
    SqlGeometryBuilder gb = GetGeometryBuilder();
    WriteGeometryPolygon(shape, gb);
    SqlGeometry geom = gb.ConstructedGeometry.MakeValid();
    return ConvertToSqlGeography(geom);
}

The last line call a method the where the SqlGeometry object is converted to a SqlGeography object by writing it out as a string and letting the SqlGeography parse that string, like so:

return SqlGeography.STPolyFromText(geom.STAsText(), 4326);

As you can see generating valid SqlGeography object can be quite a heavy operation.Depending on how large your dataset is you may have to let the computer crunch it for about an hour (which is what my 1 CPU laptop took to crunch 500MB of shapefiles), which is still managable. The above code is available in the SVN trunk at svn://svn.reimers.dk/sqlspatial

Google Maps for Mobile Phone

Tue, 14 Jun 2011 18:55:43 GMT

Since Google does not to my knowledge provide an API for putting map applications on your mobile phone I did some coding around theStatic Maps API that they do make available.

On browsers that support html input fields of type image (<input type="image" />) it is possible to move around the map by clicking the map image. Unfortunately not many mobile phone browsers support this, so I added some navigation buttons with the map.

Since mobile browsers don't have very advanced JavaScript capabilities all map movement is done server side using C#. This of course means that you don't get any AJAX features and that you will need to download a lot of map images (I tried to make the map start at your IP's geographic location, but this is not 100% accurate). On the plus side though you do get the possiblity of determining your location while you are on the move and can send this to other people in a text message.

Update

I have continued the development of the mobile application, which has brought some huge speed increases in the position calculation.

On the UI side I have added some movement buttons and a local search feature. The local search uses the map center as the search origin. If your map doesn't start in the location you think you are in then you can use this feature to search for your location (either as city or address). Once you have set your location you can start to search using the local search, and the search results will be the ones close to your position.

I use this feature to find restaurants or shops when I'm on the move. Another practical use of this is to send an SMS to someone (or more) with your position. Click on a search result and have your phone send out the link to the current page to someone using the phone browser's own 'Send Page' functionality.

Translate Resource Files

Tue, 14 Jun 2011 18:46:17 GMT

Today at work I was faced with the problem of checking if an application properly supported localization. I wrote a quick console application that takes the file location of the resource file that you want to translate and spits out a bunch of resource files in languages supported by Google Translation. Quite simple, but quite practical if you need to check if your UI works in different languages.

This example assumes that your base resource file is English, but you can change that in the code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Globalization;
using System.Web;

namespace ResourceFileTranslater
{
	class Program
	{
		static void Main(string[] args)
		{
			if (args.Length == 0) { return; }
			string saveFileFormat = args[0].Replace(".resx", ".{0}.resx");
			XmlDocument resDoc = new XmlDocument();

			foreach (string language in supportedCultures)
			{
				try
				{
					CultureInfo ci = CultureInfo.CreateSpecificCulture(language);
					string saveFile = string.Format(saveFileFormat, ci.Name);
					resDoc.Load(args[0]);
					XmlNodeList dataNodes = resDoc.SelectNodes("//data");
					foreach (XmlNode node in dataNodes)
					{
						foreach (XmlNode childNode in node.ChildNodes)
						{
							if (childNode.NodeType == XmlNodeType.Element)
							{
								if (!string.IsNullOrEmpty(childNode.InnerText))
								{
									childNode.InnerText = Reimers.Google.Translator.TranslateText(childNode.InnerText, CultureInfo.CreateSpecificCulture("en"), ci, "http://www.url.com", "LongGoogleKey");
									childNode.InnerText = HttpUtility.HtmlDecode(childNode.InnerText);
								}
								break;
							}
						}
					}
					resDoc.Save(saveFile);
				}
				finally
				{
					Console.WriteLine(language);
				}
			}

		}

		private static string[] supportedCultures = new string[] { 
            "sq", "ar", "bg", "ca", /*"zh",*/ "hr", "cs", "da", "et", /*"fil",*/ "fi", "fr", 
            "gl", "de", "el", "he", "hi", "hu", "id", "it", "ja", "ko", "lv", "lt", 
            /*"mt",*/ "pl", "pt", "ro", "ru", "sr", "sk", "sl", "es", "sv", "th", "tr", "uk", "vi" };
	}
}

Of course, using an automatic translation tool, should not be used for release code. Make sure you check the translation before you release your application.

Google AJAX Search API

Tue, 14 Jun 2011 18:35:53 GMT

Google has released a new AJAX search API for implementation in web pages. The AJAX API allows for web, local, video and blog searches. Google still has their Web API where you can search web pages through their webservice, but I'm guessing this is going to be phased out now.

Google is now spearheading AJAX for all sorts of web need. You probably already know about Google Maps (if not have a look around this site), and the new AJAX search API seeks to integrate with this service to allow users to do local searches. Many map developers have been begging Google for a geocoding service, and it seems like this new service can provide both geocoding and display search result locations. I'm not sure that the latter actually qualifies as reverse geocoding, but it certainly makes it easy to plot your search results on the map - which probably also means that we will see ads on the maps soon. For now the local search seems to work best with U.S. locations, but this should be rectified soon.

Conspicuously the AJAX search API does not provide searches for news, images or other of their specialized searches.

The API can be used to draw an AJAX search box, but more interesting is the ability to search programatically through the different search classes: GwebSearch(), GlocalSearch(), GblogSearch() and GvideoSearch(). The results can then be displayed in a customized and integrated manner on the web page.

A simple search can be done like so:

JavaScript

var s;

function DoSearch(query) {

s = new GwebSearch();
s.setSearchCompleteCallback(this, WriteSearch);
s.execute(query);
}

function WriteSearch() {
if (s.results.length == 0) { alert("No results"); }
else { for (var i = 0; i < s.results.length; i++) { alert(s.results[ i ].title); } }
}

The search string is passed to the DoSearch function, which in turn queries Google. The cool thing to notice is that users don't have to wait while the function gets the results. A search complete handler can be specified and this function will be called when the results arrive. In the example I merely loop through the results and display a message box, but in a real life implementation the results would be sent to a page element to be displayed.

I have also made a more complex implementation of the API that uses the local search function (for the moment this only works in the U.S and Canada). The local search function can help you find the nearest Chinese restaurant or can geocode and address simply by typing it into the search box. See the demo here.

A big thank you to Google for sharing their innovations with everyone.

Basic Callback Functionality Using ICallbackEventHandler

Tue, 14 Jun 2011 18:32:42 GMT

ASP.NET 2.0 introduces the ICallbackEventHandler (System.Web.UI.ICallbackEventHandler) to allow asynchronous communication with the server. Callbacks differ from Postbacks in that only the ViewState and user defined information is sent back to the server. This is in effect the core of the AJAX craze, to request only the relevant data from the server so as to make loading faster and to avoid having to reload the page every time you want to page through a large dataset or give the user constantly updated news via a news feed.

With the increased access to broadband it might not seem necessary to restrict the data exchange, which is definitely true for many pages that are more or less static. But for more dynamic pages it definitely makes sense.

This tutorial explains how to use the ICallbackEventHandler which is a simple to implement tool for adding callbacks to pages and controls. For more advanced AJAX implementation have a look at ASP.NET AJAX (previously Atlas). Both of these solutions from Microsoft rely on some lengthy JavaScript to handle the callbacks. Actually callback (or AJAX) is created using JavaScript and the XmlHttpRequest in JavaScript, so sometimes it makes more sense to create your own lightweight AJAX handler using native JavaScript and couple them with some Generic Handlers (.ashx files). But this is for more advanced users.

The easiest way to get AJAX on your page is to use the ICallbackEventHandler. The ICallbackEventHandler interface can be used by any control that implements the System.Web.UI.ICallbackEventHandler, and so is not restricted to web pages only. The following code shows the how to implement the most basic callbackhandler for a webpage.

public partial class callback : System.Web.UI.Page, ICallbackEventHandler
{
	string strResult = string.Empty;

	protected void Page_Load(object sender, EventArgs e)
	{
		string strScript = string.Format(@"
function ServerRequest(arg,ctx) {{ {0}; }}
function GetResult(result, context) {{ eval(result); }}
function GetError(result, context) {{ alert(result); }}
", Page.ClientScript.GetCallbackEventReference(this, "arg", "GetResult", "ctx", "GetError", false));

		Page.ClientScript.RegisterClientScriptBlock(this.GetType(), "CallbackFunctions", strScript, true);
	}

	string System.Web.UI.ICallbackEventHandler.GetCallbackResult()
	{
		return strResult;
	}

	void System.Web.UI.ICallbackEventHandler.RaiseCallbackEvent(string eventArgument)
	{
		strResult = "document.getElementById(\"divResult\").innerHTML = \"Callback made at: " + DateTime.Now.ToUniversalTime().ToString() + " (Universal Time)\"; ";
	}
}

This code implements the ICallbackEventHandler on a web page. In the page's load event the clientside scripts are added to the page. These could also be written directly to the HTML markup except for the GetCallbackEventReference, which is not something that is easily known in advance, so I strongly recommend creating the clientscript in the code behind using the ClientScriptManager.

We will get to the clientside script in a little while. To finish up the serverside part of the ICallbackEventHandler I will just point you to the two methods that do are the actual ICallbackEventHandler implementation. The first one I will mention is System.Web.UI.ICallbackEventHandler.RaiseCallbackEvent which takes a single string argument. The fact that the method only takes a single string means that you must decide on a proper serialization from the client to the server and then create the necessary objects on the server from the serialized information. In this example I don't use the argument for anything, but you would probably like to. This method has changed from the beta2 version to the release version of the .NET Framework and is now a void (Sub in Visual Basic). This means that you must handle the data here but cannot return it directly to the page. Instead I assign it to the page variable strResult. This variable is taken by the System.Web.UI.ICallbackEventHandler.GetCallbackResult method and sent to the page, so it doesn't need to do anything except return the strResult variable.

The result of the callback is sent back to the clientside GetResult function. I have made it simple here and made a function that simply evaluates the result sent from the server This works well in this scenario since the server generated valid JavaScript code and so can be used directly. If the server generates data in another formation you will of course have to write a function to handle the formatted data.

If you are implementing the ICallbackEventHandler on a custom control make sure that it inherits from System.Web.UI.Control or from a class that is derived from there.

4D Maps with Silverlight

Mon, 13 Jun 2011 18:57:50 GMT

The OverlayManager for the Bing Maps Silverlight control now lets you define time to your overlays through the VisibleTimeRange property, which means that it will filter out any overlays who are marked with a time range outside the visible time range. (By default overlays are always visible.)

Adding time as a dimension to a map adds a lot of interesting opportunities for map applications. Displaying markers has always been the main focus, but with interactive web applications being considered as required it becomes still more difficult to view any kind of data outside of its temporal context. This is why I added the VisibleTimeRange property to the OverlayManager. This provides an easy way to ensure that the data displayed is not outdated or not in the future based on the viewing context, i.e. showing you what is considered active content for your view.

Before we go further I should mention that the OverlayManager now defines an attached property, which allows you to easily hook it on to your map directly in XAML, like so:

XAML

<UserControl x:Class="MapApp.MainPage"
			 xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
			 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
			 xmlns:ve="clr-namespace:Microsoft.VirtualEarth.MapControl;assembly=Microsoft.VirtualEarth.MapControl"
			 xmlns:reimers="clr-namespace:Reimers.Silverlight.VirtualEarth;assembly=Reimers.Silverlight">
	<Grid HorizontalAlignment="Stretch"
		  VerticalAlignment="Stretch">
		<ve:Map x:Name="map"
				LogoVisibility="Collapsed"
				NavigationVisibility="Collapsed"
				ScaleVisibility="Collapsed">
			<reimers:OverlayManager.Manager>
				<reimers:OverlayManager />
			</reimers:OverlayManager.Manager>
		</ve:Map>
	</Grid>
</UserControl>

Once that's done you are ready to work with time enabled overlays. The Reimers.Silverlight.TimeRange class (actually struct) defines a From and To property that lets you define a time range. The OverlayManager defines a TimeRangeProperty as an attached property, which means that you can attach it to any DependencyObject. Like so:

FrameworkElement marker = new Ellipse();
DateTime markerFromDate = new DateTime(2009, 7, 1);
DateTime markerToDate = markerFromDate.AddMonths(1);
OverlayManager.SetTimeRange(marker, new TimeRange(markerFromDate, markerToDate));

If you attach a time range to an overlay the OverlayManager will read it and determine if the overlay is to be displayed depending on what time range is defined as visible for the moment.

To make it easy to define a time range in the UI, the Reimers.Silverlight assembly also comes with a TimeRangeSlider control (templatable, of course), which is basically a slider, except that it has two markers, high and low, that you can drag.

XAML

<UserControl x:Class="MapApp.MainPage"
			 xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
			 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
			 xmlns:ve="clr-namespace:Microsoft.VirtualEarth.MapControl;assembly=Microsoft.VirtualEarth.MapControl"
			 xmlns:local="clr-namespace:VirtualMap.Controls"
			 xmlns:controls="clr-namespace:Reimers.Silverlight.Controls;assembly=Reimers.Silverlight"
			 xmlns:reimers="clr-namespace:Reimers.Silverlight.VirtualEarth;assembly=Reimers.Silverlight">
	<Grid HorizontalAlignment="Stretch"
		  VerticalAlignment="Stretch">
		<ve:Map x:Name="map"
				ScaleVisibility="Collapsed"
				NavigationVisibility="Collapsed"
				LogoVisibility="Collapsed">
			<reimers:OverlayManager.Manager>
				<reimers:OverlayManager />
			</reimers:OverlayManager.Manager>
		</ve:Map>
		<controls:TimeRangeSlider x:Name="timeBar"
								  HorizontalAlignment="Left"
								  VerticalAlignment="Bottom"
								  SelectedRangeChanged="timeBar_SelectedRangeChanged" />
	</Grid>
</UserControl>

private void timeBar_SelectedRangeChanged(object sender, EventArgs e)
{
	OverlayManager.GetManager(map).VisibleTimeRange = timeBar.SelectedRange;
}

As you can see in the code above, when the SelectedRange property is updated (fx, by dragging the range slider) the map's visible time range updates as well and the markers are redrawn.

An example app could for example be an Elvis spotting map, which plots the locations where Elvis has been spotted. Without a visible time range property for the map the application would either have to display all Elvis sightings (I'm sure there are quite a few), or have to make multiple lookups every time the user changed the current view range, which means that you would lose the ability to analyze based on surrounding data.

Another more practical application would be to hook it up to your Google Analytics account. Then you can display visitor locations with time awareness and quickly change between different time periods for comparison (so many ideas, so little time).

Remember, this is still in beta, so some feature may not be fully implemented. For example, databinding is still in development. But please don't let this stop you trying it out and sending some feedback.

Consuming REST Services from Silverlight

Mon, 13 Jun 2011 18:55:38 GMT

Silverlight has a relatively advanced network stack which allows you to perform GET and POST requests to both the same domain or other domains, provided they expose a cross-domain policy (crossdomain.xml or clientaccesspolicy.xml). With Silverlight 3 it is possible to perform other request types, such as PUT and UPDATE.

In Silverlight 2 the HTTP stack used the browser HTTP stack, but in Silverlight 3 you can opt in to the client HTTP stack. To create a HTTP request using the client HTTP stack you must use the WebRequestCreator, like so:

HttpWebRequest req = (HttpWebRequest)WebRequestCreator.ClientHttp.Create(
new Uri("http://ajax.googleapis.com/ajax/services/search/web?q=silverlight&v=1.0&key=yourGoogleKey"));

This allows you to access REST services directly from the Silverlight application, making it easy to create mashup type applications. In this video Google engineer talks about the Google AJAX Search API being accessed from Silverlight. I hadn't found any implementations online, so I added it to the Reimers.Silverlight assembly. It supports web search, news search, image search, video search, local search and blog search.

The Reimers.Silverlight library currently includes support for two REST based services, Google's AJAX Search API and Panoramio's image search. The REST services are integrated into the static Services class, and can be accessed by passing the search parameters and a callback delegate to be invoked when the response is returned. The following shows an example request

private void ProcessWebSearch(SearchResult<WebSearchResult> results)
{
foreach (WebSearchResult item in results.Data.Results)
{
// Process search result
}
}
Reimers.Silverlight.Services.WebSearch("silverlight", "your Google Key", null, ProcessWebSearch);

Remember to display proper credits when you display results from a REST service that is not your own.

Starting XMPP with .NET

Mon, 13 Jun 2011 18:53:47 GMT

If you haven't heard about XMPP before then I suggest that you find out what it is. In short, it's the protocol that Google Talk uses. But XMPP is so much more, and if you want to integrate real-time communication in your application, for example for multi-user collaboration, like Google Wave, then XMPP is a good option.

XMPP and Jabber hasn't really been adopted by the .NET community yet. The most well known XMPP project for .NET is jabber-net which is a very complete library, but I wasn't too happy with the way it uses string parsing everywhere. For that reason, and for a bit of self-learning I decided to sit down and read the specifications and get to work on implementing my own library. Mind you it is still very much in development, but I thought I would put it out there and get feedback.

The main class is the XmppClient class that will allow you to connect to an XMPP server an send and receive messages. Simply create an instance of the XmppClient class and establish a connection with your JID.

XmppClient communicator = new XmppClient();
communicator.Connect("jabber.id@domain.com", "password");

The Jabber ID will most likely be your GMail account. Though if you want to start to integrate XMPP into your own applications I would recommend setting up your own server (Openfire is a good free choice).

One of the most difficult things about setting up the connection is finding out which server address to connect to. The built in DNS features in .NET don't support resolving SRV records, so I'm using the DnDNS library. It's not completely finished but it's the best option for the moment. Implementing it is in no way intended to violate any copyrights.

Establishing the communication is only half the story, but it's the story so far. Once you have the connection then you are ready to send and receive data back and forth. When an incoming stanza arrives it will trigger an event, so before calling the Connect method as shown before it is a good idea to have registered your event handlers, especially for the different stanza received events.

XMPP is about connecting to a federated network, so you may be receiving stanzas from many sources and they may be coming in fast (I don't think the Openfire server was named by accident). The XmppClient triggers the events in threads so as not to stop receiving stanzas while one is being processed, so you should take care to make your application thread safe.

The X in XMPP is for extensible (as everywhere else) and XMPP lets you transmit pretty much anything as extensions to the base stanzas (the basic XML structure for transmitting data in XMPP). The IContentStanza interface defines the core interface for an extensible stanza, and the three "base" stanzas, The PresenceStanza, MessageStanza and InfoQueryStanza implement it. This means that you can find all extended content in the ExtendedContent property. Since there is no real limit on what can be added as extended content, your application will need to pick out and parse what it needs and here LINQ for XML is your friend.

Apart from stabilizing the core communicator this is the part that is going to be under development and extension in the future. The idea is to develop built-in support for the most common extensions, but leave the extensibility open so you can fit it into your own application.

As always, the code is available for your private or commercial use. It is copyrighted, so don't remove the credits and pass it off as your own.

Testing Your Web Application With IIS Express and Unit Tests

Mon, 13 Jun 2011 18:50:38 GMT

After I read Scott Hanselman's blog post where he mentions how to launch IIS Express from the command line I thought it should be simple to integrate that into my tests so I can test the full path of requests hitting the web application. You will need to install Web Matrix in order to have access to IIS Express. Rumor has it that it will be available as a separate install later, but for now Web Matrix is a requirement. You can download it here.

In this post I'll go through the steps I went through to get it running. I'm not saying this is the best way, but it works for me.

It's one thing to get IIS Express running from the command line, but you need to have some content to serve for the requests in your tests. So you will need to make sure that the web application is built when you run your tests, so you don't test against an old version. The simple wey to do this is to add a reference, but that isn't really practical, since you don't need to reference the web site, you just need to make sure that it is built and running when you are sending your requests. A better way is to specify project dependencies, by right clicking on your test project in the Solution Explorer. Select the Project Dependencies line. This should bring up a dialog with your test project at the top and the other projects in your solution below. Check the box next to your web application to indicate that your test project depends on it. This will ensure that the web application is built before your test project.

If you are developing a normal web site it should be no problem with a normal build, but if you are using an MVC application, you will need to package it as part of the build. By default the projects created in Visual Studio have Build as their default build target (if you don't know what a build target is search for 'msbuild targets' in you favorite search engine, it's beyond the scope of this post to explain it). You will need to change your project file so that it builds and then packages the MVC application. The easiest way to do this is to right click your MVC project and select Unload Project. The right click the grayed out project and select Edit. This should bring up your build file, which is just XML. The root node should look something like this:

XML

<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

As you can see the default target is Build. Change the line so it looks like:

XML

<Project ToolsVersion="4.0" InitialTargets="Build" DefaultTargets="Package" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

The Package target is the target that is responsible for packaging the application. It is declared in some default target files for ASP.NET MVC that you don't need to worry about. What you have just done is change the default build behavior of the MVC application so that it builds the application and then packages it.

By default the MVC application will create a folder with the files that go into the package at /obj/[Debug|Release]/Package/PackageTmp relative to your MVC project's root. If you change that location you will need to change the location for your tests accordingly.

While you are configuring your build file you may want to add a property to prevent the package file from using tokens to replace your connection strings. These are added so that you can deploy your site with configurable parameters, but are a pain when you run the package files directly. This is certainly going to be a problem if you are using config file tranformations. To prevent the package from containing tokens instead of your connection strings, add the following to your build script properties:

XML

<AutoParameterizationWebConfigConnectionStrings>false</AutoParameterizationWebConfigConnectionStrings>

These files are the ones that you want to use as your site and have as the root for your IIS Express instance that your tests will be targeting.

To get your instance of IIS Express running and targeting the newly built site you will need to start it up and specify the root of the site as well as which port you want to use. This is what Scott Hanselman explained how to do from the command line in his post. Here we will be doing it as part of the test setup.

The following code samples use NUnit as the test framework, but there is not blockin you from using the xUnit or MsTest frameworks. The code to launch IIS Express is the same.

In your test setup method you will want to start a thread which runs the IIS Express process. I do it like this:

private Process _iisProcess;

[TestFixtureSetUp]
public void Setup()
{
	var thread = new Thread(StartIisExpress) { IsBackground = true };

	thread.Start();
}

private void StartIisExpress()
{
	var startInfo = new ProcessStartInfo
		{
			WindowStyle = ProcessWindowStyle.Normal,
			ErrorDialog = true,
			LoadUserProfile = true,
			CreateNoWindow = false,
			UseShellExecute = false,
			Arguments = string.Format("/path:\"{0}\" /port:{1}", AppLocation, Port)
		};

	var programfiles = string.IsNullOrEmpty(startInfo.EnvironmentVariables["programfiles"])
						? startInfo.EnvironmentVariables["programfiles(x86)"]
						: startInfo.EnvironmentVariables["programfiles"];

	startInfo.FileName = programfiles + "\\IIS Express\\iisexpress.exe";

	try
	{
		_iisProcess = new Process { StartInfo = startInfo };

		_iisProcess.Start();
		_iisProcess.WaitForExit();
	}
	catch
	{
		_iisProcess.CloseMainWindow();
		_iisProcess.Dispose();
	}
}

What the code does is start up a separate thread for IIS Express and the start the web server process. There are a couple of things to note. The first is the arguments passed to the process start. The path is the path to your packaged MVC files mentioned above, so you will need to set the AppLocation variable. The port is the port number you will be running the site on, it can be set to any valid port number (like 12345). Another thing is the location of IIS Express. It gets installed as part of Web Matrix as a 32 bit application. This means that if you are running on a 64bit system, you won't be using the same path as a 32 bit system. Fortunately the ProcessStartInfo class gives you easy access to these folder locations in the EnvironmentVariables property.

All there is to do now is to start up the web server, which the test setup will now do. The server should now be running in a window which lets you track request (not strictly necessary when testing). The web server will now respond to incoming requests at the localhost address on your chosen port as if you were running a 'real' IIS server. This means that you can send HTTP requests to it from your tests, like so:

[Test]
public void WhenRequestingDefaultPageThenGetResponseWithCode200()
{
	var request = (HttpWebRequest)WebRequest.Create("http://localhost:12345/default.aspx");

	var response = (HttpWebResponse)request.GetResponse();

	Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}

Once you have finished your tests you will want to shut it down again, which you can do like this:

[TestFixtureTearDown]
public void Teardown()
{
	_iisProcess.CloseMainWindow();
	_iisProcess.Dispose();
}

When the web server was started we created a reference to it, which makes it easy to shut it down and dispose of it. That's all there is to running your tests against an IIS Express instance.

Flocking and Clustering Using the K-Means Algorithm

Mon, 13 Jun 2011 18:46:00 GMT

I was looking around for an algorithm that would give me an outline of points in a dataset. Normally it would be a good idea to cluster the points and calculate the convex hull of the points in each cluster. The main problem with this approach is that most clustering algorithms require that you define a set number of clusters you want to create for your data points. But in you are working with points that are not evenly distributed or are bunched into natural clusters with a different count than what you specify then your clusters don't give a realistic picture of your data.

The solution is obviously to cluster the points in a cluster near their own location and create new clusters if the existing clusters are irrelevant for the current point, because it is too far away. In this post I am going to show how I modified the k-means clustering algorithm to create clusters at my points. Once the points were clustered it was a simple case of calculating the convex hull of each cluster.

The basic k-means algorithm clusters n observations into k clusters with each point belonging to the cluster with the nearest mean location, where n and k are defined before the clustering. The modified version here uses the same approach except it adds a requirement that a cluster cannot be further away than d, so that the algorithm can be said to cluster n observations into x clusters, where a cluster mean is not further than d from an observation, where only n and d are defined.

Where the k-means algorithm uses a defined k amount of clusters this approach starts by using the location of the first observation as the location of the first cluster. While looping through the dataset the location of the current observation is used to find the first nearest cluster that is not further from the observation than the max distance d. The observation is then assigned to this cluster. When reaching the end of the data set all clusters with no points assigned to them are removed and for the rest the mean location of the cluster is recalculated. The process is then repeated except that from the second iteration and on the existing clusters are used as initial clusters. This does not mean that a new cluster cannot be created because a new mean is beyond the max distance d from a given observation, but it will become increasingly unlikely.

The speed up the process I used the Task Parallel Library so the observations are clustered in parallel. This may cause an extra iteration or two because some clusters may be created in parallel and therefore be close to each other. This is the reason that empty clusters are removed at the end of each iteration. The iterations continue until the new cluster means are the same as the previous iteration.

The clustering is performed using the following code to get the initial center assignments.

private static IDictionary> GetCenterAssignments(
	IEnumerable points,
	IEnumerable centers,
	double maxDistance,
	Units distanceUnit)
{
	var centerAssignments =
		new ConcurrentDictionary>(
			centers
			.Where(x => x != null)
			.Distinct()
			.Select(x => new KeyValuePair>(x, new ConcurrentBag())));

	Parallel.ForEach(
		points,
		point =>
		{
			var closestCenter = (from centerPoint in centerAssignments.Keys
									let distance = MapCalculations.Distance(centerPoint, point, distanceUnit)
									where distance < maxDistance
									select centerPoint)
											.FirstOrDefault();

			if (closestCenter != null && centerAssignments.ContainsKey(closestCenter))
			{
				centerAssignments[closestCenter].Add(point);
			}
			else
			{
				centerAssignments.TryAdd(point, new ConcurrentBag { point });
			}
		});

	return centerAssignments.Where(x => x.Value.Count > 0).ToDictionary(x => x.Key, x => (IList)x.Value.ToList());
}

The center assignments are then moved to the new mean location for the contained coordinates.

private static IList GetNewCenters(IDictionary> centerAssignments)
{
	var newCenters = new List();

	Parallel.ForEach(
		centerAssignments.Keys,
		new Action(
			center =>
			{
				var averageLng = centerAssignments[center].Average(x => x.Longitude);
				var averageLat = centerAssignments[center].Average(x => x.Latitude);

				var newCenter = new Coordinates(averageLat, averageLng);
				newCenters.Add(newCenter);
			}));

	return newCenters;
}

As mentioned above the process is repeated until the new mean centers for the clusters are equal to the previous ones. This means that the points have been placed in the nearest cluster and the clusters are no longer moving.

Note, the code above uses the ICoordinates interface, which basically exposes Latitude and Longitude.

To make sure there are no cross thread issues when running the clustering tasks in parallel I make use of the new thread safe collections from .NET 4, so this code won’t run in .NET 3.5.

Moving to Silverlight Maps

Mon, 06 Jun 2011 09:23:43 GMT

As I have written about previously I have made a decision to move on from AJAX map applications and start working with Silverlight. I have chosen to go with the Silverlight Map Control from Microsoft. The control is still CTP, but it does seem fully functional. There are some things features missing, and the access to the full Silverlight UI engine means that while you can do whatever you want, you have to code most of it yourself.

The price to be paid for moving to Silverlight is that you can't use Google as a map supplier. As for Microsoft, then I can't get a full understanding of the license terms to use their map tiles. Since the map control allows custom tiles, I decided to load them from OpenStreetMaps. They let you use them for free, but I'm sure they won't mind a donation.

One of the first features I felt was missing, was the way overlays were handled. In the Reimers.Silverlight library you can find the OverlayCollection class, that acts as an intermediate layer between your application and the map.It does this mainly by defining several overlay layers to ensure that large overlays do not cover smaller ones, even though they are added later (i.e. have a higher z-index). It helps you define the properties for the overlays. Map overlays rely on a number of attached properties which are probably not the first thing you learnt when started on Silverlight.

Another missing feature was KML support. KML is now the standard for serializing and exchanging geographic information. The Reimers.Silverlight library allows you to read and write KML information through the OverlayCollection class. To fully support KML data, I had to define the KmlOptions class which holds the default overlay information and can be attached to any DependencyObject. Further there is the KmlStyle class that stores style information from a KML definition and can attach the relevant Silverlight styles to the DependencyObject.

Last, but not least, the assembly also includes access to geocoding, reverse geocoding and routing operations from public sources through the Services class. Microsoft offers the same operations for pay, these operations are offered for free.

SQL Server 2008, KML and Shapefiles

Sun, 05 Jun 2011 20:15:28 GMT

This weekend I had a look at SQL Server 2008 to find out about the new spatial functions and types that it includes. It has taken some time for Microsoft to include spatial capabilities, but it is here now, and it looks great.

To test out what speed I could get out of a SQL Server Express setup I wanted to take my existing shapefiles from my current SQL Server 2005 which has been extended with my own spatial types and functions. These are based on KML definitions, which I quickly found out is not supported by SQL Server 2008. I guess I should have known that, and I cannot blame Microsoft for going with OGC standards. I looked around and found the SQL Server 2008 Spatial Tools as well as an extensive set of shapefile from GADM. I tried to upload a couple of the country shapefiles using the uploader tool from Morten Nielsen.

While uploading I kept getting errors about the shapefiles not being properly formatted. As I assumed the shapefiles were correct, I wanted to have a look at the source code for the uploader, but sadly that is not available. So I thought I would start an open source shapefile reader and uploader. Actually I have already released the source code for a shapefile to KML converter, so it is merely a question of adding the ability to convert KML objects to SQL Server 2008 compatible object. The Microsoft.SqlServer.Types namespace contains theSqlGeography class, which is the CLR type that can be uploaded to the database.

I found that one of the main reasons the previously mentioned uploader failed so frequently was that shapefiles seemed to be read the wrong way round. A major restriction in SQL Server 2008 is that shapefiles cannot be larger than a hemisphere. Depending on how you view a shape, you can say that it either contains a certain area, or contains the whole globe except that certain area. As mentioned, the size cannot be larger than a hemisphere, but SQL Server doesn't seem to be able to be able to, by itself, determine which is the smaller area. (There is another restriction, that polygon boundaries cannot cross themselves). As I was able to read from various internet sites (does anyone have a good book on the spatial functions of SQL Server 2008?) the points of a polygon have to be read counter-clockwise, so as to keep the interior of the shape on the left of the border. KML files don't have this restriction. So even though the shapefile to KML converter is able to read shapefiles and create KML objects, these objects cannot be read directly into a SqlGeography object. However I found that reversing the points of a polygon made it possible to create a SqlGeography object that can be inserted into the database.

I have added a new project to the SVN trunk which contains the source code for reading Shapefiles and KML files and creating SqlGeography objects - you can find it at svn://svn.reimers.dk/sqlspatial.

The project uses the GoogleMap assembly to handle KML objects. If you read shapefiles, use the Shapefile class and fetch the included ShapeRecords from the ShapeRecord property. KML files can be read using the Reimers.Map.GoogleOverlayCollection. The records can be read from the collection itself. I have chosen to keep this as a separate project as it requires .NET Framework 3.5 SP1, and I cannot add that as a requirement for the map control at present.

I'm not going to go into too much detail about code, as this is still a work in progress. I had a chance to test it on the entirety of the GADM dataset. Trawling through the 244 shapefiles (>500MB of data) took around an hour on my old laptop, which seemed fair to me. This means I can discontinue the SQL Server 2005 Integration project, as soon as I have created an upgrade path. I don't kid myself about the SQL Server 2005 Integration project, it is fast, but it probably cannot compete with a dedicated spatial engine.