Clouds can Spread …

The ash cloud started out in Iceland where it eventually moved and spread out to cover the UK and Europe, even some reports say that it got as far as Canada. Grounding planes and crippling transport for a number of weeks.

This got me thinking, not all clouds are nice white clouds, some are evil volcanic clouds that actually do more harm than good especially when not contained to where we want them to be.  (the ash cloud would have been fine if it had not strayed into European airspace).

The same is true in your applications, when developing for the cloud you can find that the cloud can end up spreading throughout your applications and if your not careful touching every section of code you write.

This can then be a real problem when the underlying SDK to interact with the cloud changes and you have to go through all of your application looking where to refractor and make changes.

Contain Your Cloud

To combat this I contain any code that will access the cloud within few specialised “cloud” projects within my application.

This means that throughout the application if I want to access the Azure message queue, I call a single method from one of my cloud project that will return me a message, meaning that no other part of you application needs to know how to deal with the cloud.

Added Benefits

Containing your cloud not only allows you to keep to the DRY principles and code re-use it, makes maintenance of your application easier and can reduce code smells. One of the biggest benefits is that it also allows you to easier test your applications without having to rely on the cloud or developer fabric, as you can create your cloud methods in a mock-able way.

This isn’t great as you don’t want everyone knowing about your internal system and especially the URL of one of your web roles, with which they could do who know what.

This can easily be fixed with this patch for WCF that will now show the expected endpoint. This endpoint is actually the address of the load balancer that will then forward your request to an web role.

Once the patch is download and installed you can then simply add the following xml snippet into your behaviour for the WCF service:

With this snippet added into your web.config you can browse back to the previous service and will now see that the correct URL is displayed for your service.

I have recently been asked a number of questions both internally and from customers about when you will be billed for Azure usage.

As Eric Nelson’s recent post describes, if you suspend a service you will still be paying this is due to the fact you will still have your application deployed on the server ready to start again. With your application utilizing the server others will not be able to provision its usage.

When you select delete you will stop paying as you will no longer have servers provisioned for your application and the recourses will be freed up for others to use.

Memcached is a distributed cache used to help speeding up large scale web applications by taking pressure off the database. Memcached is used by many of the internets biggest sites, including Twitter, Wikipedia, and YouTube to name just a few.

A distributed cache is one of the things that I’ve been hoping to see released for Windows Azure for quite a while, and I am hoping that AppFabric Caching will make the move to the cloud in the coming year. However, until that happens I was determined to find a way to get a distributed cache and this great Windows Azure Memcached sample showed me how.

“Brad Fitzpatrick, I love your ass!"

Sorry, I just couldn’t help myself. I found this great quote on in the Caching Story page of the Memcached wiki.

After installing and setting up Memcached you will be able to cache any data, including data that is retrieved from your database so that the next time you need it you can get it from cache and not need to re-query your database. Therefore, reduce the pressure on the database and earning the love of our DBA.

Azure Memcached sample

You can download the sample code for Windows Azure from the codeplex download page you then need to download a Windows friendly version of Memcached (here is where I got mine). With the sample code from codeplex just add the memcached exe to your worker roles. You will now be able to run the sample code either on the dev fabric or in the cloud.

One thing to watch out for, the Memcached exe’s seem to take an age to get up and running. I actually left mine to set up overnight (glad I’m still on a CTP account) as the Memcached Worker Roles were all showing busy for a long time, during which the sample would not work correctly.

After the servers have started up you can easily set and retrieve values from the cache as shown in the screenshots below.

Under the hood

Now that we have got our distributed cache working lets have a look how the sample code works.

Memcached Server

On the server side we set up an internal endpoint that will be used to connect the clients to the Memcached server. When the node is created we launch the Memcached server that we downloaded before, passing in the nodes cache size and endpoint as arguments.

          string arguments = 	"-m " + cacheSize +
				" -l " + endpoint.Address +
				" -p " + endpoint.Port;

            ProcessStartInfo startInfo = new ProcessStartInfo();
            startInfo.CreateNoWindow = false;
            startInfo.UseShellExecute = false;
            startInfo.FileName = "memcached.exe";
            startInfo.WindowStyle = ProcessWindowStyle.Hidden;
            startInfo.Arguments = arguments;

This node is now exposing a Memcached server, which can be used by the client nodes to cache data.

Utilizing Memcached – Clients

The client node in the example looks a lot more complicated that it actually is. The most important part is adding the configuration for our memcached client within the settings tab of the node.

Then comes the interesting bit using the Enyim caching library (others libraries are available) and creating a MemcachedClient object, allowing us to get and set objects in the cache.

Each time a new MemcacedClient is needed the program loads a configuration based upon the data within the settings (picture above) and then loops through all of the instances of memcached servers nodes that we are running in our hosted service getting their endpoint, to add this to the configuration of available servers.

            _endpoints = new Dictionary();

            foreach (var endpoint in RoleEnvironment.Roles[_memcachedRoleName].Instances)
            {
                foreach (var epi in endpoint.InstanceEndpoints)
                  {
                    if (epi.Key == "memcached")
                    {

                        _endpoints.Add(epi.Value.RoleInstance.Id.ToString(),epi.Value.IPEndpoint);
                    }
                }
            }

Now that the connection to our memcached nodes we can use the caching library to set and get values in the cache.

AzureMemcached.Client.Store(StoreMode.Set, key, value);          

AzureMemcached.Client.Get(key);

With this code we now have distributed caching on Windows Azure, and can use the example to build out much bigger applications.  

The natural progression from here was to get the code working from within a web or worker role instance hosted within the cloud. Using a web role, I used the OnStart method to set up a IServiceManagent class that could then be shared with the remainder of the classes within the role.

When tracing the hosted services from either the OnStart method or from within default.aspx.cs I could successfully print out all of the services within my account. However, when I set up a WCF REST service to return these hosted services in an array I started to get an error with the connection to the Management API. Giving a argument error stating that a property with the name “httpRequest” is not present. This was even happening when I was using exactly the same code as I was elsewhere.

After a push in the right direction from Simon Davies we found out  that the issue was in the WCF REST service already having an OperationContext from the original REST call before calling out to the Service Management API. This handy blog post, help solve all my issues, all that was needed was to create a second OperationalContext before calling the management service.

using(new OperationContextScope((IContextChannel)WebRole.serviceManagement))
{
  var hostedServices = WebRole.serviceManagement.ListHostedServices(subscriberID);

  foreach (HostedService service in hostedServices)
  {
    Trace.TraceInformation("Hosted Service: {0}", service.ServiceName);
  }
}
            

Ta dar … everything now works, we can use the above code snippet to retrieve back all our hosted services by calling our own web role.

Eye On Earth is a new application that provides air and water quality to more than 500 million people across Europe, using both measurement stations and modelling data, along with social observations from people around not only Europe, but the whole world.

This is one of the first applications to be built and run on Windows Azure and SQL Azure allowing near real-time data to be presented at air stations and in a Europe wide air quality model.

Environmental issues, Bleeding Edge Solution

Eye On Earth uses a number of bleeding edge technologies including Windows Azure, SQL Azure and Silverlight to bring a rich user experience that can be scaled on demand whilst still processing all the near real-time data.

One of the first things that you notice when you visit Eye on Earth is how great it looks- it really shows off what you can do with Silverlight and the latest Bing Maps Silverlight Control.

Windows Azure is at the core of Eye On Earth and is not only used to host the front end, but also on the back end sits on it too – with a number of worker roles that are used to process compute intense tasks, without effecting the performance of the front end.

Eye on Earth is heavily focused on data, all of which is stored and queried using SQL Azure, along with this the application has large volumes of data being entered at near real-time from the European Environmental Agency.

But I Don’t Have Silverlight …

Eye On Earth will still love you. While it was primarily designed to be run as a Silverlight application, but with not everyone being able to use this technology, a parallel site has also been created using ASP.NET (along with some blood, sweat and tears). The ASP.NET version not only allows you to run the application anywhere – it gives you the vast majority of the features you would see in the Silverlight version.

Along with ASP.NET the team utilized jQuery and the recently released Microsoft AJAX Content Delivery Network (CDN) to give all the rich user interactions without the power of Silverlight. When being re-directed to this version of the site you can hardly tell the difference between what you would have experienced in Silverlight and what you see in ASP.NET

Quick Tip – turn off Silverlight and check it out.

Lighting up Europe

Eye on Earth has a number of “light maps” that are overlaid on a Bing Map controls to show a visual representation of both the air and water stations as well as user feedback.

Different coloured dots are used to visualise where the measurement stations are and allows users to zoom in and view more fine grained data.

User feedback, for both air and water quality can be turned on and off at the top of the map control and will show a colour coded display of where users have rated the air or water in a given location. These user ratings are coloured differently to the official readings, so that the user will be able view the social rating of given areas.

As more users rate the air and water quality around the world the light map will automatically re-generate and show an updated view of the social ratings.

Each of the light map tiles in the application have been generated using using a number complex algorithms, and processed on Azure worker roles before being stored in blobs for retrieval when a user views the site. This makes it much easier to create the tiles in parallel over multiple nodes, ensuring that the process is both fast and efficient.

Modelling Air Quality

You can also see air quality models put together by the European Environmental Agency on an hourly basis. With this model, users can be given a overview of the quality of the air around Europe from either a heat map, or placing push pins.

As this data arrives it is directly input into SQL Azure and processed, and passing the a number of compute tasks to Azure worker roles so that the images are generated correctly and displayed in the fastest possible manner.

Station to Station

As you zoom into the map within the Eye on Earth application measurement station appear as icons. Round icons represent water stations and the square icons with a wind-sock represent the air stations.

You can mouse over or click on individual measurement stations, to view more in depth data about the given station. On the left hand side the station will show the official EEA rating and a break down of the measurements that give the station this rating. On the right social user ratings are displayed, giving an idea of how the general public rate the air or water quality at the given location.

As you can see in the image this station has a good user rating, but a rating of very good from the EEA.

As more users rate the station or the air in surrounding areas, the word cloud on the left will update giving an idea of keywords that users have selected when rating the quality at this location.

Eye on Earth also displays historical water data on the water stations, allowing the user to see what the station has measured the water to be like over the past couple of years. Along with a yearly update seasonal updates will be displayed, showing how the current water season has been for the location.

We Live in a Social Environment

It’s not just the EEA that get to put data on the site – as a user, you can rate the quality of air and water in your region too.

Rating a location is simple, all you need to do is identify how good the air or water quality is at this location and then select one or more rating keywords that best describe the area.

Sharing the data at a given location can be via a number of social networks, including Facebook, Twitter and Live Spaces.

I Can See my House from Up Here

Viewing the air data within Europe is not restricted only to the measurement stations within the application. Using the air model we allow users to place pushpins anywhere on the map, and if this is within the data areas for the model it will return what the quality of air is at that location.

The user will then be able to rate or share the information at this pushpin location as if they had viewed the data from a measurement station.

If the pushpin is placed outside of the air model for Europe no air data will be retrieved, however you will still be able to rate the air at the location, using the rate button. This allows something that doesn’t exit anywhere else – a worldwide air quality indication generated by users.

Data On the Go, via SMS

Users wont always be in the comfort of their own home or workplace  and therefore not always have access to a computer – and so it was a logical step to build a mobile solution into the application for the user to get data on the move.

Eye On Earth has a SMS service that allows users to text in their location, and retrieve an automated response showing what the quality is like at the given location. 

Je Ne Comprends Pas

It’s always easy to forget that English isn’t everyone’s first language – and over the next couple of weeks we’ll be adding 24 additional languages to the Eye On Earth site, enabling more and more individuals to benefit from the site via Silverlight, ASP.NET and SMS.

Eye on Earth will automatically detect your locale as it loads, and switch not only to the correct language but also giving you localized map controls.

And Finally…

Eye on Earth is a stunning, elegant and full featured application that allows businesses, governments and individuals from around Europe to view and rate the air and water quality where they are, using the latest technologies such as Azure and Silverlight. All of this data contributes to the biggest, most up-to-date Europe wide map of air and water quality publicly available on the Internet.

Go to www.eyeonearth.eu and try the site yourself.

The total cost of ownership (TCO) and return on investment (ROI) calculator allows end users and organisations see how much money they are likely to save / spend when migrating or developing an application on the Azure Platform.

The TCO & ROI Calculator takes the user through a number of steps, starting with a number of questions about the application. Selecting such things as the type of business / application, use of SQL Azure and how you expect to see the service fluctuate over time.

Answers from this questionnaire can then be used to calculate an estimate on the total cost and nodes needed for the application.

This can then be seen over a number of years to see what the cost would be over a longer time period as well as a comparison to how much would be spent hosting the application on premise.

Check out the TCO & ROI Calculator here.

The article “Deploying to the Cloud as Part of Your Daily Build” can be found here.

You can also find out more information and subscribe to the UK MSDN Flash here.

In good presentation style (or was that lack of time) my presentation contained minimal text and relied on my presentation skills and images to convey the message, so here is a quick run down of my presentation:

Cloud Platform – Windows Azure is a cloud based platform for built from commodity machines, that allows you to develop and deploy, scalable, available applications in Microsoft’s  data centres across the world.

Utility Computing – Much like your gas and electricity you are only charged for what you consume, therefore giving a very small barrier to entry and making it ideal for small to mid-size businesses. When you need more compute power all you have to do is spin up more servers, no more buying servers based on your peak load, just scale and pay for what you use.

Web & Worker Roles – This is possibly my favourite slide, and where I think the whole concept of Windows Azure falls into place for everyone. Azure is build up on virtual machines consisting of worker and web roles I like to think of these much like the Barista and coffee makers in a coffee shop. Your order is taken by the Barista / web role and passed to the back end coffee makers / worker role for your order to be completed. If at any time either of these “fall over” there will be another role to step into its place and continue the order.

Azure Storage – There are three types of Azure storage Blobs, for binary data, Queues for queuing up messages to be processed by the worker roles, and tables which are flat file databases which will easily scale as needed.

Relational Database – although Windows Azure ships with table storage, you can also use a relational database structure using SQL Azure and SQL Management studio.

Developer Experience – Windows Azure offers a familiar developer experience for any .NET developer with plug-in’s for Visual Studio and a local developer fabric to develop on dev machines. Azure also offers SDKs for both Java and php.

Thanks for everyone who attended the talk and I hope you enjoyed it as much as I did. You can find more information about Windows Azure at http://www.microsoft.com/windowsazure/ along with the SDK and developer tools that you will need to get started.