Now that I’m blogging again I thought it’d be time for a little update.

We’ve been getting some questions along the lines of “I am hosting a service as xyz.svc in IIS and have changed the config to use on of the Service Bus bindings, but the service never gets called?”

That’s right. It doesn’t. The reason for that is that we don’t yet have WAS/IIS integration for any of the Service Bus bindings in the November 2008 CTP. Enabling the WCF WAS activation scenario that puts the NetTcpRelayBinding and friends on par with their WCF siblings is on our work backlog for the next major milestone.

It’s worth considering for a moment what that integration requires. Fundamentally, all of the Relay bindings replace the local TCP or HTTP listener with a listener that sits up in the cloud and services then connect up to that listener to create an inbound route for received messages. That’s similar to how local services interact with WCF’s shared TCP listener or HTTP.SYS, but there are quite a few important differences. First, all Relay listeners need to acquire and present an Access Control token when they start listening on the Service Bus. In contrast, the local listener facilities are ACL’d using the local or domain account system and use the Windows process identity to decide on whether a process may or may not listen on a particular port and/or namespace. Second, since the actual listener is off-machine, we need to spin up the connection as the IIS/WAS host spins up and need to make sure that the connection is kept alive and aggressively reconnects when dropped for any reason. That’s something you don’t really have to worry much about when the listener sits right there on the same machine as your own service and the connection is a named pipe. Third, the local listeners listen on a particular host address and port; the Relay listeners listen on a leaf of a namespace tree and that namespace may be shared amongst many listeners living on a multitude of different machines in different locations.  Fourth,   ... well you get the picture.

Bottom line: Not having support for WAS activation and xyz.svc service endpoints is by no means an oversight. It’s on the list.

The MSDN Developer Center for .NET Services is the first stop to go to for technical information on the Service Bus, the Access Control Service and the Workflow Service.

There quite a bit of documentation for “my” feature area, the .NET Service Bus, including description of all the bindings and most of the object model surface area. Since we had quite a bit of object model churn up until a few weeks before PDC as we’ve exploded the former, singular RelayBinding into two handful of WCF-aligned bindings, the reference documentation isn’t yet in the familiar MSDN reference format and also doesn’t yet work with Visual Studio’s “F1”. We’re obviously going to address that in the next major milestone now that the dust is settling a bit and the programming model is already quite a bit closer to what we want it to be for our “V1” release.

In the sea of PDC 2008 announcements you may have missed the following two signficant developments:

• .NET Services for Java
• .NET Services for Ruby

For the past 2 months our team has worked very closely with our partners at Schakra on the Java SDK parts and with ThoughtWorks on the Ruby parts. These are the first baby steps and these two SDKs cover only a small subset of the capabilities of the .NET SDK so far. That's merely a function of when we started with these projects and how far we've gotten with the required protocol support; we want and we will take this a lot further over the next development milestones. In the end, the .NET Services fabric ought not to care much what language the senders and listeners are written in and what platform they run on. We're building a universal services platform. We're taking Java and Ruby very seriously and have a few more platforms on the list for which we want to add explicit support.

If you want to try out Windows Azure, or .NET Services, or SQL Services, you need an access code. How do you get one? By signing up here.

"Yes, I did that, but I didn't get a code, yet!"

We're onboarding new accounts slowly but steadily so that we do a controlled scale ramp-up. PDC attendees (having signed up with the same LiveID that they used to register for PDC) will be getting their access codes first, everyone else will be getting their access codes after that. We're a bit conservative with the onboarding waves and closely monitor the overall utilization once we allow a new wave in. So if you attended PDC and don't have a code it may still take a few days for us to give you one depending on when you signed up. If you didn't attend PDC we're going to try giving you codes as excess capacity permits.

Our team's PDC talks are online on Channel 9:

• Azure Platform Overview (John & Dennis): http://channel9.msdn.com/pdc2008/BB01/
• .NET Services Architecture Overview (John & Dennis): http://channel9.msdn.com/pdc2008/BB02/ 
• .NET Access Control Service Intro (Justin): http://channel9.msdn.com/pdc2008/BB55/
• .NET Access Control Service Drilldown (Justin): http://channel9.msdn.com/pdc2008/BB28/
• .NET Workflow Service Intro (Moustafa): http://channel9.msdn.com/pdc2008/BB27/
• .NET Service Bus Intro (Clemens): http://channel9.msdn.com/pdc2008/BB38/
• .NET Service Bus Drilldown (Clemens): http://channel9.msdn.com/pdc2008/BB12/
• Diagnostics, Logging, Debugging (Steve & Mark): http://channel9.msdn.com/pdc2008/BB39/ (the actual talk starts ~11min into the recording)
• Designing Apps for Scale (Max): http://channel9.msdn.com/pdc2008/BB54/ 

We've got a discussion forum up on MSDN where you can ask questions about Microsoft .NET Services (Service Bus, Workflow, Access Control): http://social.msdn.microsoft.com/Forums/en-US/netservices/threads/

According to recent traffic studies, the BitTorrent protocol is now responsible for roughly half of all Internet traffic. That's a lot of sharing of personal photos, self-sung songs, and home videos. Half! Next to text messaging, Instant Messaging applications are the social lifeline for our teenagers these days – so much that the text messaging and IM lingo is starting to become a natural part of the colloquial vocabulary everywhere. Apple's TV, Microsoft's Xbox 360, and Netflix are shaking up the video rental market by delivering streamed or downloadable high-quality video and streams on YouTube have become the new window on the world. Gamers from around the world are meeting in photorealistic virtual online worlds to compete in races, rake in all the gold, or blast their respective Avatars into tiny little pieces.

What does all of that have to do with Web 2.0? Very little. While it's indisputable that the Web provides the glue between many of those experiences, the majority of all Internet traffic and very many of the most interesting Internet applications depend on bi-directional, peer-to-peer connectivity.

These familiar consumer examples have even more interesting counterparts in the business and industrial space. Industrial machinery has ever increasing remote management capabilities that allow complete remote automation, reprogramming, and reconfiguration. Security and environment surveillance systems depend on thousands of widely distributed, remotely controlled cameras and other sensors that sit on street poles, high up on building walls, or somewhere in the middle of a forest. Terrestrial and satellite-based mobile wireless technologies make it possible to provide some form of digital connectivity to almost any place on Earth, but making an array of devices addressable and reachable so that they can be integrated into and controlled by a federated, distributed business solution that can leverage Internet scale and reach remains incredibly difficult.

The primary obstacle to creating pervasive connectivity is that we have run out of IPv4 addresses. There is no mere threat of running out, we're already done. The IPv4 space is practically saturated and it's really only network address translation (NAT) that permits the Internet to grow any further. The shortage is already causing numerous ISPs to move customers behind NATs and not to provide them with public IP address leases any longer. Getting a static public IP address (let alone a range) is getting really difficult. IPv6 holds the promise of making each device (or even every general-purpose computer) uniquely addressable again, but pervasive IPv6 adoption that doesn't require the use of transitional (and constraining) tunneling protocols will still take many years.

The second major obstacle is security. Since the open network is a fairly dangerous place these days and corporate network environments are often und unfortunately not much better, the use of Firewalls has become ubiquitous and almost all incoming traffic is blocked by default on the majority of computers these days. That's great for keeping the bad guys out, but not so great for everything else – especially not for applications requiring bi-directional connectivity between peers.

Since these constraints are obviously well-known and understood there is a range of workarounds. In home networking environments the firewall and NAT issues are often dealt with by selectively allowing applications to open inbound ports on the local and network router firewalls using technologies like UPnP or by opening and forwarding port by ways of manual configuration. Dynamic DNS services help with making particular machines discoverable even if the assigned IP address keeps changing. The problem with those workarounds is that they realistically only ever work for the simplest home networking scenarios and, if they do work, the resulting security threat situation is quite scary. The reality is that the broadly deployed Internet infrastructure is optimized for the Web: clients make outbound requests, publicly discoverable and reachable servers respond.

If your application requires bi-directional connectivity you effectively have two choices: Either you bet on the available workarounds and live with the consequences (as BitTorrent does) or you build and operate some form of Relay service for your application. A Relay service accepts and maintains connections from firewalled and/or NAT-ed clients and routes messages between them. Practically all chat, instant messaging, video conferencing, VoIP, and multiplayer gaming applications and many other popular Internet applications depend on some form of Relay service.

The challenge with Relay services is that they are incredibly hard to build in a fashion that they can provide Internet scale where they need to route between thousands or even millions of connections as the large Instant Messaging networks do. And once you have a Relay that can support such scale it is incredibly expensive to operate. So expensive in fact that the required investments and the resulting operational costs are entirely out of reach for the vast majority of software companies. The connectivity challenge is a real innovation blocker and represents a significant entry barrier.

The good news is that Microsoft .NET Service Bus provides a range of bidirectional, peer-to-peer connectivity options including relayed communication. You don't have to build your own or run your own; you can use this Building Block instead. The .NET Service Bus covers four logical feature areas: Naming, Registry, Connectivity, and Eventing.

Naming

The Internet's Domain Name System (DNS) is a naming system primarily optimized for assigning names and roles to hosts. The registration records either provide a simple association of names and IP addresses or a more granular association of particular protocol roles (such as identifying domain's mail server) with an IP address. In either case, the resolution of the DNS model occurs at the IP address level and that is very coarse grained. Since it is IP address centric, a DNS registration requires a public IP address. Systems behind NAT can't play. Even though Dynamic DNS services can provide names to systems that do have a public IP address, relying on DNS means for most ISP customers that the entire business site or home is identified by a single DNS host entry with dozens or hundreds of hosts sitting behind the NAT device.

If you want to uniquely name individual hosts behind NATs, differentiate between individual services on hosts, or want to name services based on host-independent criteria such as the name of a user or tenant, the DNS system isn't an ideal fit.

The .NET Service Bus Naming system is a forest of (theoretically) infinite-depth, federated naming trees. The Naming system maintains an independent naming tree for each tenant's solution scope and it's up to the application how it wants to shape its tree. 'Solution' is a broad term in this context meant to describe a .NET Service Bus tenant – on the customer side, a Service Bus application scope may map to dozens of different on-site applications and hundreds of application instances.

Any path through the naming tree has a projection that directly maps to a URI.

Let's construct an example to illustrate this: You design a logistics system for a trucking company where you need to route information to service instances at particular sites. The application scope is owned by your client, 'ContosoTrucks' which has a number of logistics centers where they want to deploy the application. Your application is called 'Shipping' and the endpoints through which the shipping orders are received at the individual sites are named 'OrderManagement'. The canonical URI projection of the mapping of New York's order management application endpoint instance into the ServiceBus Naming system is
http://servicebus.windows.net/services/contoso/NewYork/Shipping/OrderManagement/

The significant difference from DNS naming is that the identification of services and endpoints moves from the host portion of the URI to the path portion and becomes entirely host-agnostic. The DNS name identifies the scope and the entry point for accessing the naming tree. That also means that the path portion of the URI represent a potentially broadly distributed federation of services in the Naming service, while the path portion of a 'normal' URI typically designates a collocated set of resources.

There is no immediate access API for the Naming system itself. Instead, access to the Naming system is provided through the overlaid Service Registry.

Service Registry

The Service Registry allows publishing service endpoint references (URIs or WS-Addressing EPRs) into the Naming system and to discover services that have been registered.

The primary access mechanism for the Service Registry is based on the Atom Publishing Protocol (APP) allowing clients to publish URIs or EPRs by sending a simple HTTP PUT request with an Atom 1.0 'item' to any name in the naming tree. It's removed by sending an HTTP DELETE request to the same name. There is no need to explicitly manage names – names are automatically created and deleted as you create or delete service registry entries.

Service discovery is done by navigating the naming hierarchy, which is accessible through a nested tree of Atom 1.0 feeds whose master-feed is located at http://servicebus.windows.net/services/[solution]/. Any publicly registered service is accessible through the feed at the respective location.

In addition to the Atom Publishing Protocol, the Service Registry also supports publishing, accessing, and removing endpoint references using WS-Transfer and the Relay service will automatically manage its endpoints in the Service Registry without requiring any additional steps.

The Service Registry is an area that will see quite significant further additions over the next few milestones including support for service categorization, search across the hierarchy, and support for additional high-fidelity discovery protocols.

Connectivity

The core of the connectivity feature area of the .NET Service Bus is a scalable, general-purpose Relay service. The Relay's communication fabric supports unicast and multicast datagram distribution, connection-oriented bi-directional socket communication and request-response messaging.

Towards listening services the Relay takes on the same role as operating-system provided listeners such as Windows' HTTP.SYS. Instead of listening for HTTP requests locally, a relayed HTTP service establishes an HTTP listener endpoint inside the cloud-based Relay and clients send requests to that cloud-based listener from where they are forwarded to the listening service.

The connection between the listener and the Relay is always initiated from the listener side. In most connection modes (there are some exceptions that we'll get to) the listener initiates a secured outbound TCP socket connection into the Relay, authenticates, and then tells the Relay at which place in the naming tree it wants to start listening and what type of listener should be established.

Since a number of tightly managed networking environments block outbound socket connections and only permit outbound HTTP traffic, the socket based listeners are complemented by an HTTP-based multiplexing polling mechanism that builds on a cloud-based message buffer. In the PDC release the HTTP-based listeners only support the unicast and multicast datagram communication, but bidirectional connectivity is quite easily achievable by pairing two unicast connections with mutually reversed client and listener roles.

A special variation of the bi-directional socket communication mode is 'Direct Connect'. The 'Direct Connect' NAT traversal technology is capable of negotiating direct end-to-end socket connections between arbitrary endpoints even if both endpoints are located behind NAT devices and Firewalls. Using Direct Connect you can start connections through the Relay and 'Direct Connect' will negotiate the most direct possible connectivity route between the two parties and once the route is established the connection will be upgraded to the direct connection – without information loss.

With these connectivity options, the Relay can provide public, bi-directional connectivity to mostly any service irrespective of whether the hosting machine is located behind a NAT or whether the Firewalls layered up towards the public network don't allow inbound traffic. The automatic mapping into the Naming system means that the service also gains a public address and the service can, on demand, be automatically published into the Service Registry to make the service discoverable.

In addition to providing NAT and Firewall traversal and discoverability the delegation of the public network endpoint into the Relay provides a service with a number of additional key advantages that are beneficial even if NAT traversal or discoverability are not a problem you need to solve:

• The Relay functions as a "demilitarized zone" that is isolated from the service's environment and takes on all external network traffic, filtering out unwanted traffic.
• The Relay anonymizes the listener and therefore effectively hides all details about the network location of the listener thus reducing the potential attack surface of the listening service to a minimum.
• The Relay is integrated with the Access Control Service and can require clients to authenticate and be authorized at the Relay before they can connect through to the listening service. This authorization gate is enabled by default for all connections and can be selectively turned off if the application wants to perform its own authentication and authorization.

These points are important to consider in case you are worried about the fact that the Relay service provides Firewall traversal. Firewalls are a means to prevent undesired foreign access to networked resources – the Relay provides a very similar function but does so on an endpoint-by-endpoint basis and provides an authentication and authorization mechanism on the network path as well.

If your applications are already built on the .NET Framework and your services are built using the Windows Communication Foundation (WCF) it's often just a matter of changing your application's configuration settings to have your services listen on the Relay instead on the local machine.

The Microsoft.ServiceBus client framework provides a set of WCF bindings that are very closely aligned with the WCF bindings available in the .NET Framework 3.5. If you are using the NetTcpBinding in your application you switch to the NetTcpRelayBinding, the BasicHttpBinding maps to the BasicHttpRelayBinding, and the WebHttpBinding has its equivalent in the WebHttpRelayBinding. The key difference between the standards WCF bindings and their Relay counterparts is that they establish a listener in the cloud instead of listening locally.

All WS-Security and WS-ReliableMessaging scenarios that are supported by the standard bindings are fully supported through the Relay. Transport-level message protection using HTTPS or SSL-protected TCP connections is supported as well.

If the listener chooses to rely on WS-Security to perform its own authentication and authorization instead of using the security gate built into the Relay, the HTTP-based Relay bindings' policy projection is indeed identical to their respective standard binding counterparts which means that client components can readily use the standard .NET Framework 3.5 bindings (and other WS-* stacks such as Sun Microsystems' Metro Extensions for the Java JAX-WS framework).

If you prefer RESTful services over SOAP services, you can build them on the WebHttpRelayBinding using the WCF Web programming model introduced in the .NET Framework 3.5. The Relay knows how to route SOAP 1.1, SOAP 1.2 messages and arbitrary HTTP requests transparently.

The NetEventRelayBinding doesn't have an exact counterpart in the standard bindings. This binding provides access to the multicast publish/subscribe capability in the Relay. Using this binding, clients act as event publishers and listeners act as subscribers. An event-topic is represented by an agreed-upon name in the naming system. There can be any number of publishers and any number of subscribers that use the respective named rendezvous point in the Relay. Listeners can subscribe independent of whether a publisher currently maintains an open connection and publishers can publish messages irrespective of how many listeners are currently active – including zero. The result is a very easy to use lightweight one-way publish/subscribe event distribution mechanism that doesn't require any particular setup or management.

The discussion of the close alignment between the Relay's .NET programming experience and the standard .NET Framework shouldn't imply that the Relay requires the use of the .NET Framework. Microsoft is working with community partners to provide immediate and native Relay support for the Java and Ruby platforms of which initial releases will be available at or shortly after PDC with more language and platform support lined up in the pipeline.

The Relay provides connectivity options that allow you build bidirectional communication links for peer-to-peer communication, allows making select endpoints securely and publicly reachable without having to open up the Firewall floodgates, and provides a cloud-based pub/sub event bus that permits your application to distribute events at Internet scale. I could start enumerating scenarios at this point, but it seems like a safe bet that you can already think of some.

Find out more here:
http://www.microsoft.com/azure/default.mspx
http://www.microsoft.com/azure/servicebus.mspx

We’re thrilled to announce that the BizTalk Services “R12” Community Technology Preview (CTP) is now available for general use.

“BizTalk Services” is the code-name for a platform-in-the-cloud offering from Microsoft.  Currently in active development, BizTalk Services provides Messaging, Workflow, and Identity functionality to enable disparate applications to connect quickly and easily.
Combined together in an integrated offering, these capabilities deliver a Service Bus architectural pattern that is immediately usable by applications that need to connect across the Internet. 

Many enterprises employ the ‘Enterprise Service Bus’ pattern to interconnect disparate systems within an organizational domain. Built on Microsoft platform technology, an ESB might include building blocks such as Windows Server, Active Directory, BizTalk Server, as well as the Windows Communication Foundation and Windows Workflow Foundation technologies included in the .NET Framework.  “BizTalk Services” extends the concept of an ESB to truly exploit the Internet, for instance by exposing individual service endpoints in a secure fashion or by selectively federating elements of distinct identity systems to facilitate cross-company collaboration.
 
For ISVs and Solution Providers creating specialized business solutions that enable collaboration and information exchange across increasingly mobile and distributed work-forces, “BizTalk Services” provides the cloud-based platform building blocks to create sophisticated (Internet-) Service Bus solutions with broad reach that could otherwise only be realized by operating dedicated Data Centers of significant complexity – which is often out of reach for both, ISVs and their customers.

Major Changes

With the release of BizTalk Services “R12”, developers must update all clients and SDK installations to the new release. 

New in R12 - Workflow

The most exciting new capability we’ve added in the “R12” CTP is Workflow. These new cloud-based Workflow capabilities enable ‘service orchestration’ from the cloud.  This specialized cloud-based, or hosted, Windows Workflow Foundation runtime can orchestrate services that connect to systems in your enterprise, or to systems running anywhere on the Internet via Web services messages.  This new power and capability will enable an entirely new set of application scenarios, and we’re very excited to see what people will do with it.

In the SDK you will find  samples showing how to create and control Workflow instances hosted on the BizTalk Services cloud, including a sample Workflow implementation that monitors the availability of a website and fires multicast events into the service bus indicating the state.

New in R12 - Identity

For R12, the BizTalk Services Identity Service has been expanded and enhanced to enable more flexibility for scenarios demanded by our customers.  R12 introduces a new approach for creating, viewing, and managing access control rules. This approach relies on a few key principles outlined below:

• Every Identity Service account owns a Security Token Service (STS).
• An STS is composed of one or more scopes.
• A scope contains zero or more access control rules.
• An STS owner can grant another Identity Service account permission to edit the access control rules in a scope

A practical illustration to clarify:. The Messaging Service owns an STS whose root scope is http://connect.biztalk.net/services/. When you create a new account (newaccount) in the Identity Service, the messaging service creates a new scope http://connect.biztalk.net/services/newaccount. The Messaging Service then grants (newaccount) the permission to create access control rules in that scope. Any communication endpoints hosted there can thus be secured by the owner of the scope.  Rules from R11 accounts have been migrated to the “root” scope of the new account.

On the protocols front, we’ve added several new capabilities for ‘REST’ services. We now support integration with Windows Live ID and have added RFC2617 Basic and HTTPS/Client Certificate support for acquiring security tokens using simple HTTP GET requests.

New in R12 - Messaging

Connectivity Modes

The most fundamental new feature area in the Messaging service are the new ‘connectivity mode’ settings on the RelayBinding. Before this release, BizTalk Services clients and listeners always required outbound TCP ports 808 and 818 to be available for connecting to the BizTalk Services cloud for all connection modes except the clients of a listener running with ConnectionMode.RelayedHttp. 

In this release we are introducing three different connectivity modes: Tcp, Http, and AutoDetect. The connectivity mode can be set on a static property of the RelayBinding. The Communication\ExploringFeatures\ConnectionModes\Multicast sample shows how. For clarity: ‘Connection Mode’ defines the type of end-to-end connection that is to be established through the Relay. ‘Connectivity Mode’ defines how a particular endpoint connects up to the Relay.

The ‘Tcp’ connectivity mode is the most efficient one and works as in previous releases. The ‘Http’ mode is new. It creates a volatile FIFO buffer for messages in the BizTalk Services cloud and polls for messages using HTTP ‘parked requests’.  The Http model exhibits delivery latency characteristics similar to Tcp mode, albeit with slightly higher bandwidth consumption on idle connections. The ‘AutoDetect’ mode will check whether TCP connectivity is available and will choose ‘Tcp’ if that’s the case and ‘Http’ otherwise.

The new HTTP-based connectivity option is only effective for the RelayedOneway, RelayedMulticast and RelayedDuplex connection modes. RelayedDuplexSession, HybridDuplexSession, and RelayedHttp (listener only) still require TCP connectivity at this time. 

Transport Credentials and Unauthenticated Access

Also, in the “R12” release, the model for specifying the client credentials for the Relay has now been closely aligned with the standard WCF client credentials model. Instead of picking and instantiating token providers, there is now a TransportClientEndpointBehavior that holds all credential information and credential types. The samples in the Communication\ExploringFeatures\RelayAuthentication of the SDK download clarify the use of this new behavior.

We have added a pair of ‘WebNoAuth’ samples which introduce a new capability that we had a lot of requests for: Unauthenticated client access. When registering a service listener you can now explicitly waive the authentication requirement for clients connecting to your service. This is very useful in Web scenarios where you want to enable any HTTP client to connect to your service and don’t want them to authenticate in any way. For the time being we suggest that you always use this new  unauthenticated access mode for RelayedHttp services until we release the update for the ‘Web’ client authentication capability.

For R12, we have omitted the ‘Web’ (REST) samples for Relay authentication since that area is undergoing some substantial protocol changes.  The update for this will be released soon. In the interim, existing applications that were built on a prior release of the BizTalk Services SDK to use the authentication technique shown in the R11 ‘Web’ sample must be modified to use unauthenticated access as shown in the new ‘WebNoAuth’ sample.  

Give it a try

The new BizTalk Services “R12” CTP is online and available now for your use.  The SDK is available at http://labs.biztalk.net. If you already have an account for BizTalk Services, your accounts and settings have been migrated to the new environment. If you don’t have an account yet, just sign up, download the SDK, and get started creating the new generation of connected applications.  

The BizTalk Services CTP will be switched from the "R11" to the "R12" release starting in about 30 minutes and we expect to have a 2 hour time window (1400h-1600h PT/2300h-0100h UTC) where existing service accounts are being rolled over to the new release. We're expecting to be done with the migration by 1600h. Once the migration is done we'll give you an update on what's new in R12.