Margriet and Nikander on SharePoint

PowerPivot: brief discussion

zondag 2 mei 2010 11:32:58

PowerPivot: brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 2, 2010

PowerPivot: brief discussion

Excel is a very powerful tool that can be used by end users for analyzing corporate data. Excel also allows you to import data from various data sources, such as a database. The contents of such data sources may be huge. PowerPivot integrates with both Excel and SharePoint and tries to make Excel more scalable, faster and flexible. PowerPivot for Excel literally allows you to import, filter, and sort millions of rows of data, allows you to build relationships between data coming from different data sources, and stores data in a highly compressed file stored on the client.

You can also share PowerPivot workbooks by publishing them to a SharePoint 2010 farm with the help of Excel services and SQL Server PowerPivot for SharePoint. If you do this, you can view such PowerPivot workbooks from within a SharePoint library.

SharePoint Workspace

zondag 2 mei 2010 11:32:58

SharePoint Workspace

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 2, 2010

SharePoint Workspace

The ultimate ECM solution should allow information workers to work so flexible that they can do so anywhere they want, and anyhow they want to work. The ability to work offline is an essential part if you are really serious about executing such a vision. To this end, a couple of years ago Microsoft decided to acquire a product called Groove. In SharePoint 2010, it's called SharePoint Workspace and it fits in with SharePoint Server 2010 in a very natural way. Because of the importance of offline working, we will devote a blog post to this topic that shows you the basics of working with SharePoint 2010 Workspace.

Click Start > All Programs > Microsoft Office > Microsoft SharePoint Workspace 2010. This opens SharePoint Workspace.
Then, open Internet Explorer and browse to a SharePoint site. We have chosen to open a site called TestSite. It contains a couple of documents of different types.
Click Site Actions > Sync to SharePoint Workspace. This is shown in Figure .
Sync'ing to a SharePoint Workspace causes the Sync to Computer dialog window to open. Click the Configure button shown in Figure.
This opens the Configure Settings window. You can use it to configure which lists and libraries you want to synchronize.
For example, if you click the Shared Documents library, the possibility appears to download All Content, Headers Only, or No Content.
Once you have made your choices, the synchronization continues.
If you switch back to SharePoint 2010 Workspace, you will notice that the the site you have just synchronized is displayed.
If you click on the TestSite node, you will see the offline equivalent of your SharePoint site.
You can do the same thing the other way round by dragging and dropping a file to the Shared Documents folder in SharePoint workspace. In this case, we will add a document called MyText.txt.

After some time, you will notice that the document MyText.txt is added to the Shared Documents library in SharePoint as well.

PowerShell: Interact with SharePoint in the fastest way possible

zaterdag 1 mei 2010 12:38:58

PowerShell: Interact with SharePoint in the fastest way possible

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

PowerShell: Interact with SharePoint in the fastest way possible

Windows PowerShell is taking the administrative side of the Microsoft world by storm. What it is? An extensible automation engine built on top of the .NET Framework that allows administrators to perform administrative tasks. This is typically done via cmdlets (command lets) which are specialized .NET classes implementing a particular operation. The investments in PowerShell for SharePoint 2010 have been huge: SharePoint 2007 was shipped with 182 different stsadm commands, during the beta cycle, SharePoint 2010 already has a total of 652 PowerShell cmdlets. We're absolutely convinced that everybody who is serious about his work and wants to call himself a good SharePoint administrator needs to be well versed in PowerShell. That's how important we think PowerShell is for IT Pros.

Note. If you want to use PowerShell to interact with SharePoint 2010, you need to have considerable privileges. You need to have db_owner rights for all SharePoint 2010 databases, and need to be a site collection administrator for the site collections you are interacting with.

If you're not an administrator, but a developer, we don't feel that you need to become a wizard in PowerShell before you can call yourself a good developer. However, we do think it's important to have knowledge about this topic, for two main reasons:

It may save you tons of time, because there are a lot of scenarios where PowerShell is simply the fastest way to accomplish things. During development, demonstrations of your applications for the customer, or presentations you might be giving to fellow developers, you will come across actions that need to be performed, such as enabling a feature for every site in a site collection, finding out which content databases are associated to a site collection, or exeucting a specific timer job now. You will find that PowerShell is great for stuff like that.
If you're shipping a custom application it would be a great asset if you ship it with a set of custom PowerShell cmdlets that can be used to perform custom administrative tasks specifically for your application. This allows administrators to leverage the knowledge they have about PowerShell, and gives them a consistent management experience which will no doubt make them look more favorable at your application.

In this blog post, we will show all sorts of examples that demonstrate what PowerShell can do for you and the basic knowledge you need to have to work it. First of all, if you want to start using PowerShell and interact with SharePoint 2010, you need to fire up the SharePoint 2010 Management Shell. This automatically loads all SharePoint 2010 related PowerShell cmdlets. Since the primary target audience for using these cmdlets are IT Pros, most cmdlets have a "high-level" focus that at the most fine grained level operates on SharePoint site level (SPWeb object level). The next procedure explains how to start the SharePoint 2010 Management Shell.

Click Start > All Programs > Microsoft SharePoint 2010 Products > SharePoint 2010 Management Shell.
This opens the Administrator: SharePoint 2010 Management Shell command prompt, which looks like this:

It's useless trying to memorize all 600+ cmdlets. Instead, you should memorize how to retrieve a list of all available commands:

get-command

A sample of the cmdlets you can find here are shown in the following Figure.

This is still a bit much information to digest, so you can filter it by name:

get-command get-sp*

The first SharePoint cmdlet that we will take a look at is Get-SPFarm (in case you're wondering, names of cmdlets are not case sensitive). If you want to retrieve the current farm, do this:

Get-SPFarm

This returns an SPFarm object. The command prompt displays the name of the configuration database and status indicator that tells that the farm is online. If you want help using this cmdlet, you can do the following:

get-help Get-SPFarm

You can also request help in the form of realistic examples for using the cmdlet. There are lots of cases when this form of help will give you valuable extra information, it's probably our favorite help mode.

get-help Get-SPFarm -examples

As said before, the Get-SPFarm cmdlet returns an SPFarm object. It's extremely useful to find out which properties an object of this type actually has. You can determine that via:

Get-SPFarm | Get-Member

This would be another good place to apply a filter to the property name:

Get-SPFarm | Get-Member p*

PowerShell is used to automate tasks and a command prompt is the main tool you will be working with. Nevertheless, in some cases you may prefer to interact in a more visual way. In that case, you need to install the Windows PowerShell Integrated Scripting Environment feature from Server Manager, as shown in the next procedure.

Start > Administrative Tools > Server Manager. This opens the Server Manager MMC snap-in.
Click the Features node.
Click Add Features. This opens the Add Features Wizard.
Select the Windows PowerShell Integrated Scripting Environment (ISE) checkbox.
Click Next, followed by Install, then Close.

When you are finished doing that, you will be able to take the output of a cmdlet and use the PowerShell pipestream system (using the pipe symbol or "|") to something else. In this case, we will take the output of the Get-SPFarm cmdlet and redirect it to a special gridview:

Get-SPFarm | Out-GridView -Title "Test"

In the next Figure, we show an example that's just a bit more complex than the previous example, so you get a better idea of what the tool can do for you.

You can redirect ouput results to a wide range of other destinations, such as a CSV file, to name but one:

Get-SPFarm | Export-Csv c:\temp\output.txt

One of the great advantages of PowerShell is that you can store output results in a variable for later use. In the next example, we will store the output of the Get-SPFarm cmdlet in a variable called $Farm:

$Farm = Get-SPFarm

You can take this variable, and use a pipe to determine the properties of the variable:

$Farm | select *

You can also store the call to the Get-SPFarm cmdlet itself in a variable, in which case the variable is called a script block. Then, at a later time, you can choose to execute the script block via Invoke-Command:

$scriptBlock = { Get-SPFarm }
Invoke-Command $scriptBlock

Let's move on to the next SharePoint cmdlet; if you want to look at information about all databases in the entire SharePoint farm you should do this:

Get-SPDatabase

Use the next cmdlet to retrieve all service applications in your SharePoint farm:

Get-SPServiceApplication

The following returns a list of all current web applications:

Get-SPWebApplication

This list doesn't include the web application hosting SharePoint Central Administration. If you want to have that information too, you need to pass an extra parameter:

Get-SPWebApplication -IncludeCentralAdministration

If you don't want to have a list of all web applications, but only a list containing the first five, you can do this:

Get-SPWebApplication | select -First 5

You can sort the output, based on properties such as the display name:

Get-SPWebApplication | Sort DisplayName

Or simply reverse the display name sort order:

Get-SPWebApplication | Sort DisplayName -descending

You can get at a single web application using an indexer:

$webapps = Get-SPWebApplication
$webapps[0]

Or apply more advanced filters:

Get-SPWebApplication | Where { $_.DisplayName -eq "SharePoint 80" }

You can use pipes to perform operations on a result set as well. In this case, we will display all content databases for all web applications by passing in the output containing all web applications to another cmdlet:

Get-SPWebApplication | Get-SPContentDatabase

Again, you can use pipes to find out what you can do with content databases:

Get-SPWebApplication | Get-SPContentDatabase | select *

You could also use an indexer and access a property of one of the content databases in the result set:

$cdb = Get-SPWebApplication | Get-SPContentDatabase
$cdb[0].SqlConnectionString

It requires a special syntax if you want to accomplish something similar to the previous example in one go. The @{Expression=} syntax allows you to specify in which property of a complex object you are interested in. We will also use the special $_ variable that refers to the current instance of an object in a collection. In the next example, we show you how to retrieve the connection string for every content database:

Get-SPWebApplication | Get-SPContentDatabase |
Select Name, @{Expression={$_.SqlConnectionString}}

If you want to retrieve all site collections, simply go:

Get-SPSite

If you want to retrieve a limited amount of site collections:

Get-SPSite -limit 2

You can also retrieve site collections using regular expressions:

Get-SPSite "http://sharepoint2010/(my|)" - RegEx

This returns our default site collection at http://sharepoint2010 as well as the site collection containing personal sites located at http://sharepoint2010/my.

You can apply a whatif construct to commands, which allows you to perform a dry run without actually affecting anything. For instance, the next command shows which site collections would be deleted if you would try to remove all SharePoint 2010 site collections, but it doesn't actually do it:

Get-SPSite | Remove-SPSite -whatif

If you want to loop through each item in a collection, you will need to use the special ForEach-Object construct. The following command displays a property of a sub object of each item in the result set:

Get-SPSite | ForEach-Object { $_.RootWeb.Title}

The next example gets all features for all site collections and applies a sorting based on the display names of the features:

Get-SPSite |% { Get-SPFeature -site $_ } | sort DisplayName

You can take this to the next level by getting all features for all sites within all site collections and sort it by their display name:

Get-SPSite | ForEach-Object { Get-SPFeature -Web $_.Url } | sort DisplayName

The following performs a dry run of enabling the Ratings feature for all sites within all site collections (remove the -whatif part if you actually want to do it):

Get-SPSite | ForEach-Object { Enable-SPFeature "Ratings" -url $_.url -whatif }

If you want to perform a dry run of diasling the Ratings feature for all sites within all site site collections (remove the -whatif part if you actually want to do it):

Get-SPSite | ForEach-Object { Disable-SPFeature "Ratings" -url $_.url -whatif }

The next command retrieves the titles of all sites of a given site collection:

Get-SPSite "http://sharepoint2010" | get-spweb -Limit All | Select Title

You can retrieve all sites and filter the results based on the template of a site:

Get-SPSite "http://sharepoint2010" |
get-spweb -Limit All -Filter {$_.Template -eq "ENTERWIKI#0"} |
Select Title

Alternatively, you can apply a like filter:

Get-SPSite "http://sharepoint2010" | get-spweb -Limit All -Filter {$_.Template -like "E*"} |
Select Title

If you want to know which site templates are available, do:

Get-SPWebTemplate

You can use variables to perform multiple actions, sich as retrieving all lists and list items of a root site of a SharePoint site collection, and still keep to code clean and easy to read by using multiple variables:

$site = Get-SPSite "http://sharepoint2010"
$root = $site.RootWeb
$root | GetMember (if you're not sure what it's members are)
$lists = $root.lists
$lists | ForEach-Object { $_.Title }
$pagesList = $lists["Pages"]
$items = $pagesList.Items
$items | ForEach-Object { $_.Title }

PowerShell can be used locally on the server, but you can use it to issue remote commands to (with the help of the Invoke-Command cmdlet), which is really handy if you want to retrieve information about another server in your SharePoint farm. If you want to do this, you first need to enable PowerShell remoting (by starting WinRM on each server). You can do this via the following command (just accept all options presented to you):

Enable-PSRemoting

When accessing other servers, you typically will want to have a change to provide credentials without having to hard-code them. You can do this by calling the Get-Credential cmdlet which causes the Windows PowerShell Credential Request dialog window to open. This dialog window allows you to specify a user name and a password. The cool thing is, you can assign the result to a variable and use it later when making a remote call. The next Figure shows the Windows PowerShell Credential Request dialog window.

The next command shows how to read the contents of a web.config file on a remote machine (remember: that machine needs to have WinRM enabled) and passes credentials as a part of the call:

$cred = Get-Credential
Invoke-Command -ComputerName sharepoint2010 -ScriptBlock {Get-Content
c:\inetpub\wwwroot\wss\VirtualDirectories\80\web.config} -Credential $cred

Even greater, you can run a remote command on multiple computers at the same time:

Invoke-Command -ComputerName WFE1, WFE2 -ScriptBlock {Get-Content
c:\inetpub\wwwroot\wss\VirutalDirectories\80\web.config} -Credential $cred

We're concluding this section with examples related to something that we have needed a zillion times in the past and is made real easy in PowerShell. First, we're retrieving a list of all timer jobs:

Get-SPTimerJob

Finally, we use the information found via the Get-SpTimerJob to execute a single timer job, using the following syntax:

Get-SPTimerJob [name of timer job]

In the previous example of the blog post, we're executing a timer job called job-workflow, and we're executing it now:

Get-SPTimerJob job-workflow | Start-SPTimerJob

By now, you have seen loads of examples of using PowerShell for SharePoint. We have covered the essentials you need to learn in order to become effective with it and we hope you agree PowerShell for SharePoint is a valuable addition to your bag of tricks.

Fast Search Server 2010 for SharePoint: brief discussion

zaterdag 1 mei 2010 12:38:58

Fast Search Server 2010 for SharePoint: brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

Fast Search Server 2010 for SharePoint: brief discussion

We're amazed at how many times we've been put in a position where we had to defend SharePoint search against customers, end users, and competitors. Questions like: Do you call that horrible thing a document preview? Why isn't faceted search not available out of the box, we're not going to use an open source project from codeplex in a production environment? Indexing document attachments is acting really weird, how come? Why is the number of IFilters so limited? And we could go on, because there's more. Ultimately our answer always comes down to the following: SharePoint Search is an advanced search mechanism that provides good value for your money, since you don't need to pay separate license fees. If you're not happy with it, you need to invest more money.

Additional search investments come in two flavors: there have been products that are add-ons to SharePoint search and most often concentrate on providing user interface enhancements, and there have been products that replace SharePoint search altogether. One thing we want to comment though is that high-end enterprise search features come at a high-end price. If you're wondering what exactly constitutes a high-end enterprise search system, to us it's a search solution that indexes anything you want, and as much of it as you want, while still being able to infer intelligent relations from content and metadata. Later, we will talk a bit more about the specific strength points of Fast.

We guess Microsoft realized it didn't have a real solution for the high-end enterprise search market and that that led them to the acquisition of the Norwegian company Fast, which specializes in delivering search services for the high-end market. According to Gartner, Fast is even more: it's the best enterprise search system in the world. The acquisition was announced in January 2008 and ultimately, in SharePoint 2010, this has lead to a fully SharePoint-integrated solution called Fast Search Server 2010 for SharePoint. Fast Search Server 2010 for SharePoint is an add-on, which means you can start using the normal SharePoint Enterprise Search features as long as you want and switch to Fast once you need high-end search features. You can do this by adding two new search service applications:

A service application for the FAST Query interface, which is used by each WFE.
A service application for the FAST Content Gathering interface, which is used by the SharePoint Gatherer process.

At this point we still feel that the awesome power of Fast is still not adequately translated to the SharePoint 2010 user interface, but we believe this is a matter of time. The following is a small list of things Fast is really good at:

It has a fully customizable indexing pipeline. This makes it easy to determine which components should be added to a pipeline or add your own via scripts. You can also define multiple indexing pipelines. Overall, the Fast indexing pipeline architecture gives great control over the indexing process.
It supports many different kinds of document types and document attachments.
It's able to handle very large documents as well as huge amounts of data. A Fast employee once told us proudly that they had just finished indexing Norway, which equated to 40 Yottabyte.
It's great at inferring hidden relationships based on content and metadata.
The scale-up and scale-out facilities of Fast are quite fantastic.
Visual Best Bets. A normal Best Bet is shown if an end user enters a particular search term. Visual Best Bets also allow you to define best bets that are visual, such as an image, or a video.
And, oh yeah, it offers a Preview function in SharePoint 2010 (although we're still not convinced it's a great one).

You can add multiple Fast servers to your SharePoint 2010 farm. Based on preliminary information, you can expect each node to handle up to 30 million documents. Once you have set up the required infrastructure, you can go ahead and create a new site based on the FAST Search Center site template.

Word Services: brief discussion

zaterdag 1 mei 2010 12:38:58

Word Services: brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

Word Services: brief discussion

Word Services is a SharePoint service application, that can be run on one or more application servers within a SharePoint farm, that allows you to manipulate Word documents on the server without requiring the need to install Word on the server itself. Word Services also exposes the Word Services API, a managed object model, which runs on each WFE and allows developers to tap into this power via .NET languages such as C#. This allows you to initiate, stop, or monitor bulk document assembly and conversion tasks. In the back-end, SQL Server keeps track of a queue of all tasks that are either pending, in progress, or completed. The results of bulk actions are stored in a location in the SharePoint farm. Word services looks like a valuable addition to the SharePoint set of services, with a robust implementation for doing bulk operations. Having said that, in the past we have used advanced document manipulation components such as the ones from Aspose (http://www.aspose.com) that have been able to fulfil all our needs in this area.

Mobile Access: brief discussion

zaterdag 1 mei 2010 12:38:58

Mobile Access: brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

Mobile Access: brief discussion

Accessing SharePoint via a mobile device has been enhanced in several ways. First off, end users accessing SharePoint via a mobile device browser are detected automatically and redirected to a mobile page. End users can request the external URL of a SharePoint 2010 site via e-mail. End users can receive alerts in the form of instant messages sent to their mobile devices. The mobile version of the end user's personal site allows them to update the current online status, track colleagues' activitities, view RSS feeds, view the profile pages of other colleagues, view a list of colleagues and their online status. The web part framework has been updated so that it includes a mobile web part framework that allows you to build web parts that render differently on mobile pages. End users are now able to access a wide range of item types, such as Word, Excel, and PowerPoint documents (via document viewers for Word, Excel, and PowerPoint that are available out-of-the-box), view virtual lists (database tables that appear as lists on SharePoint pages), wiki pages, mobile web parts, thumbnails of pictures in a picture library, and view their personal calendar. End users can also lookup people, switch between list views, and filter list items. SharePoint 2010 supports shortcuts that provide one-button navigation for mobile users.

What is ECM?

zaterdag 1 mei 2010 12:38:58

What is ECM?

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

What is ECM?

When we first started working with SharePoint, we were working with the beta version of SharePoint Portal Server 2001 to implement an intranet for a medium-sized law firm. At that time, SharePoint consisted of a digital dashboard, the newest version of MS Search and a bunch of document management features (and a really groovy Windows shell extension that made it possible to interact with SharePoint using Windows Explorer, which we still kind of miss to this day). Our customers used SharePoint Portal Server 2001 mainly as a document management system, migrating documents from their file servers to a SharePoint repository, although it wasn't a very scalable one at the time.

After working intensively with SharePoint Portal Server 2001, we had the honour to become one of the first eleven SharePoint Most Valuable Professionals (MVP) world-wide. Imagine our surprise when we were invited to the MVP summit in Redmond and learned about the new features of SharePoint Portal Server 2003: it had arguably less document management features than SharePoint Portal Server 2001 and it's architecture had been overhauled completely, switching it's repository from a light version of the Exchange Web Storage System (yes, that was what WSS meant originally) to SQL Server. Another thing became quite clear as well: SharePoint Portal Server wasn't just a document management product, it was a portal product. That comprised a lot more. A portal offers a collection of content and application services to employees, customers, partners, and/or suppliers via internet, intranet, and/or extranet. Or to put it another way: the goal of a portal is to enhance collaboration and efficiency. A funny thing happened to end users too. Suddenly they weren't normal employees anymore, they became information workers. A typical portal product delivers on its promise by offering features in the following areas:

Legacy application life cycle. Portals may extend the life cycle for legacy applications, because they offer a centralized place where legacy application may be reused.
Consistent user interface. A consistent user interface throughout an entire portal lowers training costs and makes it easier for end users to find information, which makes end users more productive.
Web-based access. Web-based access is an ideal starting point to reach the goal to work anywhere anyhow.
Personalization. Portals typically offer personalized user experiences and content.
Customization. A portal product should be (and normally is) highly customizable according to the customer's needs.
Syndication. Portals reuse content from other locations, a concept known as syndication.
Aggregation. Closely related to Syndication, portals allow the aggregation of content and services via a universal access point.
Search. Portals offer advanced search mechanisms that allow information workers to search, at a minimum, within the portal. Usually a portal search mechanism will allow you to search other locations within the enterprise as well.
Taxonomy. Stemming from the Greek word "taxis", meaning "order", taxonomy allows you to structure unstructured or semi-structured content.
Business intelligence. Business intelligence is about handing companies advanced tooling for analyzing business data.
Integrating external content. This refers to any tooling a portal product has to integrate external content that has to stay external, but at the same time can be manipulated via the portal.
Collaboration. A portal absolutely has to provide tools that make it easier for information workers to work together and share information with each other.
Publishing. This refers to tooling offered by the portal to publish content.
Subscription. In today's world of information overload, it is essential that a portal offers some type of subscription mechanism that allows information workers to stay up to date without requiring them to log in to the portal just to check if something is new.
Workflow. Portals handle semi-structured content, and workflows related to this content (such as a simple approval workflow) is a normal part of the activities of information workers.
Business Process Integration. Business Process Integration is all about connecting enterprise applications by streamlining the transfer of business information. Business Process Integration in itself doesn't belong to the realm of a portal, but it is important that a portal is able to integrate with a Business Process Integration product. In the case of SharePoint, this product is BizTalk Server.
Multimedia distribution. Any multimedia assets owned by an enterprise can be shared via a portal. Ideally, a portal offers some specific features for working with specific multimedia types as well.
Security. Information disclosed via a portal should be done in a secure way using fine-grained authorization techniques.
Administration. Portals that offer a centralized and unified way of administrating it can cut costs tremendously.
Synergy. The term synergy may be a bit vague. It is about the whole sum (the portal) being more than all it's separate parts (the portal features). Bringing all separate portal features together as a whole adds value.

As you have seen, a document management product is a quite different beast than a portal product. In it's 2007 release, the contours of something else became visible: SharePoint Portal Server became more than a portal product: it became the first release that was able to compete on the Enterprise Content Management (ECM) market. This has even become more clear in SharePoint Server 2010. Not only does it show in the feature set that is available out-of-the-box, even the word "Portal" is dropped from it's name.

Most contemporary sources about SharePoint acknowledge that SharePoint is an ECM platform and that it's goal is to deliver ECM for the masses, but they fail to provide a good explanation of what ECM is exactly and why ECM is something more than a portal. This has lead us to believe that a discussion of what ECM is really all about would be a good idea for the blog post you are reading right now.

So: what is Enterprise Content Management?

First of all, let's be clear about this: the functionality offered by a portal has great overlap with the functionality offered by an ECM product. You're not entering a complete new realm of solutions here.

Now, let's take a step back and get down to the basics… Most companies store data in a way that matches the existing structure or architecture of business applications in use. Ultimately, this leads to a data structure that matches the structure of a company itself. The sales department then uses sales data, the software department uses data about software design, and so on. The result of this is a set of department-oriented data structures in a company. This means that data consumption is optimized for departmental use, but not for multi-department use. This leads to a data landscape that is fragmented, which in result leads to data redundancy. A prominent trait of redundant data is that it isn't synchronized. In general this means that a document that is modified in one location, won't be modified in the same document stored elsewhere, which leads to a loss of information quality. And, needless to say (but we're going to do it anyway), that's a bad thing.

To battle the loss of information quality, companies typically create procedures that need to prevent this loss. That leads to hidden costs that is the result of information fragmentation and process redundancy. It's not rare that this leads to a "fix it back" scenario, where a certain procedure ordains data to be fixed, whereas another procedure commands the contrary and results in fixing the data back to its original situation.

You can certainly say that companies are managing digital information and have been trying to do this for years. One of the things that have become quite clear is that integrating data per application is not the best long term strategy. Doing this forces companies to keep data within the company semantically consistent, which leads to sky high data maintenance costs. On the other hand, companies increasingly need to have information available within the entire company, not just within a department. Defining information metadta plays a crucial role in doing this, although it's not the only way.

EIM, which stands for Enterprise Information Management, is a framework that offers the possiblility to develop and execute an enterprise-wide strategy concerning information management. EIM is a collection of disciplines, technologies, and solutions that help to create and maintain a consistent interpretation of business data that can be consumed by everyone (users, applications, and services)! EIM focuses on structured data, the data stored in business applications, as well as unstructured data, such as Word or Excel documents, pictures, videos, paper documents. Usually this type of information is scattered across a company like a diaspora. The management of unstructured data is also known as ECM, which stands for Enterprise Content Management.

As a side note, software are usually the primary (and more suitable) consumer of structured data, humans are usually the primary (and more suitable) consumer of unstructured data.

If we need to come up with a definition of ECM, it would look something like this: ECM is an approach that tries to control all unstructured data within a company, which is accomplished by adding structure to unstructured data, most notably in the form of metadata. ECM is more than just a collectoin of products and technologies, it's also an architecture that allows companies to manage and reuse content.

As we've seen previously, the goal of a portal is to make collaboration easier and more efficient. by offering a collection of content and application services. This goal is very similar to the goal of ECM, which tries to make it possible to collaborate on a multi-departmental level, do this in a more efficient way, and offer access to unstructured data within the company This leads to a repeating pattern: usually the implementation of a portal strategy and an ECM strategy are executed at the same time using the same platform. Or, to put it in other words, most companies don't really make a difference between their portal and ECM strategies.

To debug one of the myths you may have heard surrounding ECM, your ultimate ECM goal shouldn't be to collect all enterprise content within a single, centralized system, it should be to create a uniform interface that allows you to access all information.

ECM, similar to what we saw when discussing portals, knows many facets and we will discuss them all:

DM, DM addresses document management features such as check in/check out, document security, versioning.
WCM, stands for Web Content Management, an approach to manage content for web applications. This also used to be known as: "automating the web master".
RM, stands for records management. Records management technology allows you to manage the life cycle of a piece of informaiton. RM allows you to manage content from the moment of creation, up to the moment of destruction, adressing topics like archiving, relationships between content (such as compound documents). Companies use RM to be able to abide to laws and regulations. RM can also apply to "soft information" like e-mail and instant messages.
Document capture and imaging, this facet is about processing paper documents. For instance, you can scan paper documents, have them processed by OCR (Optical Character Recognition), and store the result in SharePoint.
Document-centric collaboration, this facet focuses on sharing documents within project teams and stimulating intra-team activities. Examples of document-centric collaboration features are calendars, forums, whiteboards, instant messaging, presence awareness, surveys, and project management.
Workflows and work process management, this facet is about automating business processes. Document centric workflows like sending a document for review and then publishing them are ideal for SharePoint.
Information retrieval, also known as ESS (Enterprise Search Strategy), focuses on searching enterprise content.
ECI, stands for Enterprise Content Integration and acts as a software layer between applications and content repositories that allows companies to integrate heterogenous environments.
Federation, also known as Single Sign On (SSO), allows end users to log in only once and access every piece of information they want without ever having to log in again.
IDARS stands for Integrated Document Archive & retrieveal Systems. IDARS systems are specialized in maintaining static information like invoices and reports (documents that have been finalized, not the dynamic kind you can generate using technology such as Reporting Services).
Forms capture and processing, basically an extension to Document capture and imaging. This facet also uses recognition technologies to process information, only it goes a step beyond simple recognition. Forms capture and processing technologies recognize content parts and are able to extract key fields such as a customer ID which is compared automatically with a customer database.
DAM, Digital Asset Management, concentrates on managing rich media, particulary the image assets within an organization. A Typical DAM feature would be showing a thumbnail of an image asset in an image library, or even allowing thin clients such as a browser to perform simple graphical actions such as resizing or cropping images, batch jobs that convert a set of images to another format.
MAM, stands for Media Asset Management, a subset of DAM. MAM focuses on videa and audio rich media assets.
COLD/ERM, stands for Computer Output to Laser DIsk/Electronic Reports Management. This facet focuses on high-volume computer generated reports. COLD/ERM typically offers indexing and report mining capabilities.

Themes: brief discussion

zaterdag 1 mei 2010 12:38:58

Themes: brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

Themes: brief discussion

A theme is a thmx file that allows you to modify colors and fonts by applying the theme file to a specific SharePoint site. This doesn't give you the control found when you start adding custom CSS files, because those let you modify all sorts of page elements, such as font types, size, and spacing. On the other hand, it's not very invasive and easy to reuse. Themes can be used with Windows SharePoint Services 2010 sites as well as SharePoint Server 2010 site and affect the following UI elements:

Web part chrome.
Ribbon elements (aka, the Fluent user interface).
Pages in the _layouts folder.
The Site Actions button and menu.
Shortcut menus.
Dialog boxes.
Highlighting for bulk editing operations.

Service Applications : brief discussion

zaterdag 1 mei 2010 12:38:58

Service Applications : brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

May 1, 2010

Service Applications : brief discussion

SharePoint 2010 introduces the concept of service applications. SharePoint 2010 consists of a whole bunch of services, including Search service, Excel Calculation service, User Profiles service, Business Connectivity Services, Web Analytics, Word services, PowerPoint Broadcast service, PerformancePoint, Visio services, Access services, Managed metadata service, and Sandboxed Code service. Service applications can be either implemented as WCF services or traditional .ASMX web services and they replace the Shared Service Providers (SSP) model. Service applications are placed on SharePoint application servers which provides excellent scaling capabilities. In essence, a service application provides data or computing resources. SharePoint WFE's always communicate through service instances, which are running physical instances of service applications that can run in a dedicated application pool. Possible topology configurations for service applications are so flexible it allows you to create a services farm containing services that are reused by multiple SharePoint farms.

Note. You can even create your own custom service applications, which is a rather complicated task and before you start doing it, know that there is seldom justification for creating your own custom service application.

REST: The SharePoint RESTful interface

vrijdag 30 april 2010 12:09:58

REST: The SharePoint RESTful interface

Written by: Margriet Bruggeman, Nikander Bruggeman.

April 30, 2010

REST: The SharePoint RESTful interface

SOAP may be the de facto standard when it comes to solving problems surrounding remote communication and interoperability, but it's also true that SOAP can be quite complex and for many types of applications complete overkill. REST on the other hand, which stands for Representational State Transfer, is rapidly gaining ground as a lightweight Web service protocol that allows you to remotely manipulate resources over the Internet. Although not strictly necessary, REST is usually used in conjunction with the HTTP protocol. The exciting news is that SharePoint 2010 supports a RESTful interface, and it does this by leveraging ADO.NET Data Services.

Note. By the time you are reading this blog post, this note has hopefully become superfluous. The SharePoint 2010 product team promises that REST is enabled by default in the RTM version of SharePoint 2010. At the time of writing however, you still need to enable it by installing ADO.NET Data Services v1.5 ctp and resetting IIS. There's one strange quirk though, if you install the full version (called ADONETDataServices_v15_CTP2.exe) it always result in an access denied error and a failed installation. Instead, you need to install the run time only version called ADONETDataServices_v15_CTP2_RuntimeOnly.exe.

All REST calls to SharePoint are nothing more than HTTP methods. A typical REST interface supports the following HTTP methods:

OPTIONS, this method returns a list of all HTTP methods that are supported. The SharePoint RESTful interface doesn't support this method.
GET, this method retrieves a resource, but doesn't alter it in any way. Or to define it real precisely: the method returns a representation of the server resource and doesn't alter the server resource in any way.
HEAD, this method returns only the HTTP headers (the metadata), but not the actual resource data. The SharePoint RESTful interface doesn't support this method.
POST, this method creates a new resource on the server. You can POST to anything that represents a set. At a later time, we will create a custom list called Products. A Products list is a collection, so you would be able to POST to that.
PUT, this method creates or replaces a resource on the server.

The SharePoint RESTful interface supports another, custom, HTTP method called MERGE. You can use it if you want to avoid replacing an entire resource using PUT. For example, you may want to do this if all you want to do is replace a couple of properties of a resource. The MERGE method ensures your changes will be merged with the resource properties found on the server. The MERGE method is a custom construct by Microsoft, and it's identical to the PATCH method that hasn't been standardized yet. Once the PATCH method is standardized, Microsoft will support that too.

The examples in this section use the HTTP GET method exclusively. If you want to experiment with other HTTP methods as well, you may consider downloading the HTTP client tool cURL, which makes this easy to do. On a side note, although using cURL is very easy, installing it is not. There may be easier alternatives out there.

REST Responses can be represented via JSON or Atom (more on these response types later). The HTTP Accept header determines which content type is returned. The Accept header value for Atom is application/atom+xml, the Accept header value for JSON is application/json.

Note. In some ways, REST reminds us of WebDAV, only it's much more simple to use. This is just an observation that attempts of manipulating resources have been around for quite some time now. REST itself saw the first light of day in 2000.

Atom

Atom applies to two different standards, the Atom Syndication Format and the Atom Publishing Protocol (AtomPub). The Atom Syndication Format is an XML language used for web feeds, but this is not the standard we're interested in as it's not supported by SharePoint 2010. SharePoint 2010 does offer full AtomPub support, which is a simple HTTP-based protocol for creating and updating web resources. Sounds like an excellent match with REST, doesn't it? If you want to see examples of what Atom looks like, you will just need to follow the examples later on and you will see plenty of it.

Compared to JSON, most data services client libraries use Atom because it has richer type expression possibilities, although JSON is more compact and therefore more suitable for AJAX applications.

JSON

JSON (pronounced Jason) is a serialization format created in 2002 as a cleaner and lighter alternative to XML. It is possible to express simple as well as complex types in JSON. JSON is based on a subset of the JavaScript programming language and is very easy to parse and generate in a JavaScript programming environment. This makes JSON a well-suited, lightweight data-interchange format for browser-server communication. If you want to learn more about JSON, visit http://www.json.org/.

A JSON representation of a string is a collection of zero or more characters wrapped in double quotes. This means the JSON representation of the name William Bender looks like this: "William Bender". A JSON representation can be converted to a JavaScript (simple or complex) type using the JavaScript eval() function. You could also dynamically generate a new function that handles the conversion using the Function() JavaScript function. Both approaches are shown here:

var strJSON = "\"William Bender\"";
var strResult = eval(strJSON);
strResult = new Function("return " + strJSON);

The JSON representation of a number is identical to the number itself. For example, the JSON representation of the number 7 is 7:

var strJSON = "7";
var intResult = eval(strJSON);

The JSON representation of a Boolean is either true or false. Here’s an example:

var strJSON = "true";
var blnResult = eval(strJSON);
if ( blnResult )
{
alert(blnResult);
}

An array is an ordered collection of values. The JSON representation of an array begins with a left bracket ([) and ends with a right bracket (]). Values are separated by a comma (,). The JSON representation of a number array looks like this: [1, 2, 3]. Here’s an example:

var strJSON = "[1, 2, 3]";
var arrResult = eval(strJSON);
alert(arrResult[0]);

The JSON representation of a string array looks like this: ["Bob", "Sue"]. Here's an example:

var strJSON = "[\"Bob\", \"Sue\"]";
var arrResult = eval(strJSON);
alert(arrResult[1]);

Objects are unordered sets of name-value pairs. JSON representations of objects are enclosed by curly braces ({}). Each name is followed by a colon (:). Name-value pairs are separated by commas (,). Look at the following definition of a JavaScript object called Person containing Name and Age properties:

function Person(strName, strAge)
{
this.Name = strName;
this.Age = strAge;
}

The JSON representation of the Person object looks like this: {"Name":"Jack","Age":47}. Although you are able to use the eval() function to convert this JSON string representation to a JavaScript object containing Name and Age properties, this is more complex than converting JSON representations to simple types. Instead, we recommend you use a JSON parser to do the conversion for you. Nowadays, you can download several JavaScript frameworks that contain JSON parsers (among other functionalities). A good example of a very popular JavaScript framework that contains a JSON parser is jQuery. We definitely recommend using it.

Another argument for using a JSON parser is that the eval() function compiles and executes any JavaScript code. This makes you vulnerable ftor script injection attacks. A good JSON parser will offer safeguards against this sort of attack and thus will be safer.

Note It is possible to create complex object hierarchies via JSON because nested types are allowed.

Compared to Atom, JSON is more lightweight and very suitable for AJAX applications, but due it's lack of expressiveness it's type fidelity is less.

Working with REST

If you start working with REST in a SharePoint 2010 environment, you will directly find out that the ListData WCF services is the pinnacle of SharePoint's RESTful interface. The ListData service is virtualized, which means you can access it within every SharePoint context. To make this possible, the ListData service is stored in a virtual folder called _vti_bin. The _vti_bin folder is a virtual directory in IIS that is mapped to the [drive letter]:\Program Files\Common Files\Microsoft Shared\web server extensions\14\ISAPI folder. The name of the _vti_bin folder originates from a company called Vermeer Technologies Incorporated, which was acquired by Microsoft in the beginning of 1996. Vermeer Technologies was responsible for creating the original version of FrontPage. The _vti_bin folder is a special Common Gateway Interface (CGI )–like folder that contains server-side functionality that needs to be available within the entire virtual server. On a SharePoint server, the _vti_bin folder is the place where the default SharePoint web service files are stored.

To demonstrate the use of the ListData service, we have created a separate site collection called REST which can be reached at the following URL: http://neptune/sites/REST. If you call the ListData service within the context of the site collection, you will get a list of all lists within the root web site. The call to the ListData service looks like this:

http://neptune/sites/REST/_vti_bin/listdata.svc

The result is an XML response in Atom format that looks like this:

Now, if you would have been interested in a list of all lists in a subsite called TestSite, all you have to do is pass that context to the ListData service, like so:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc

Suppose we have created a custom list in TestSite called Products that contains the columns ID, Title, Description, and Available. We have included two test entries in the Products list called My Product and Another Product. If we want to learn more about the Products list via REST, we need to pass the list context to the ListData service, like so:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products

Sometimes MSIE is too smart for it's own good. You already know that the previous call leads to an Atom response. We're using a default installation of MSIE 8 and this is what we get:

When working with REST, it makes perfect sense to turn of the Feed Reader view so you can inspect the actual Atom response. The next procedure explains how to do this for MSIE 8.

Open MSIE 8.
Click Tools > Internet Options. This opens the Internet Options dialog window.
Click the Content tab.
In the Feeds and Web Slices section, click Settings.
Uncheck the Turn on feed reading view checkbox.
Click OK twice.

Now, if you refresh the page, the full Atom response will be displayed. In addition, we also like to use a tree web debugging tool called Fiddler (http://www.fiddler2.com). With it, you can easily collect and inspect the HTTP traffic generated by your browser, as shown in the next Figure. This comes in handy if you temporarily want to keep track of the REST calls you're making.

Note. Another interesting feature of Fiddler is that is allows you to modify and re-issue HTTP requests again. If you replace the HEADER "Accept: */*" with "Accept application/json" and execute it, you will see a REST response in JSON format.

Making a ListData call to the Products list results in an Atom response containing basic information about the Products list and the items within this list. To get an idea what a response looks like, we have included the XML bit describing the list item entry for My Product:

One of the critical pieces of information about a list entry is the element that contains a URL that you can use to further inspect this particular list entry. You can do this by issuing the following command:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products(1)

Note. Please note that the number listed in the previous command is the SharePoint 2010 list item id.

If you want to learn more about a specific column, you can add it's name to the URL:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products(1)/Title

If you want to loose the XML that surrounds the response, you can also access the text value of the property itself, like so:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products(1)/Title/$value

REST supports multiple options for presenting data. The first one that we're showing is the possibility to sort the result set based on a column name.

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products?$orderby=Title

Note. REST options are prefixed by a dollar sign ("$"). This makes it easy to distinguish them from normal querystring parameters.

The previous command also could have been rewritten as:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products?$orderby=Title asc

As you would expect, the command can be reversed like this:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products?$orderby=Title desc

You can also specify orderings for multiple columns:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products?$orderby=Title desc, ID asc

The $skip and $top parameters are ideal for implementing a paging mechanism. The $skip=[n] parameter allows you to skip the first n options, the $top=[n] parameter allows you to return the next n items. Typically, you will use this in conjunction with the $orderby parameter (appended using the & (ampersand) symbol). The following command skips the first 5 product list items (ordered by title) and displays the next one:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Products?$skip=5&$top=1&$orderby=Title

Suppose we're adding a new list called Purchases that has a lookup column called PurchasedProduct that refers to the Title column of the Products list. Using REST, you can do a lookup traversal too:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Purchases(1)/PurchasedProduct/Description

If you want to learn more about other properties of a related Product item, you can do that by appending the name of the Product list Column you are interested in:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Purchases(1)/PurchasedProduct/Description

When doing remote calls, it is usually a good idea too get big chunks of data in one go instead of using a "chatty" interface consisting of multiple calls. The $expand parameter can assist here because it allows you to embed one or more sets of related entities in the overall result set (much similar to a SQL JOIN). The next example shows how to use it:

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Purchases?$expand=Products

You can use the $metadata parameter if you want to learn what properties are available for a certain resource. If it makes it easier to understand, you can also look ath the $metadata parameter as providing WSDL for REST.

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Purchases?$metadata

The final parameter is the $filter parameter. It allows you to define filters that decrease the size of your result set. You can do all sorts of handy stuff such as comparisons (eq, ne, gt, lt, and, or, not), simple arithmetic operations (such as add, sub, mod), more advanced string operations (such as substring, startswith, trim, toupper), and data functions (such as day, month, and year). For a full list and description of all REST operations, please refer to the MSDN article "Using Microsoft ADO.NET Data Services". The next example shows how to filter the Title column, where the column value needs to be equal to the value "Product 1".

http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Purchases?$filter=Title eq 'Product 1'

Issuing HTTP GET requests using C# isn't hard to do. The next piece of code shows how to do it:

string url = " http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc/Purchases(1)";
WebRequest request = WebRequest.Create(url);
request.Credentials = CredentialCache.DefaultCredentials;

WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseContent = reader.ReadToEnd().Trim();

As the default returns XML (Atom), the string variable responseContent contains it's response in XML. You can take it, and further process it from there. Typically, you will also use the XMLHttpRequest object, jQuery framework, ASP.NET AJAX, Silverlight etc. to communicate with the SharePoint RESTful interface. Unfortunately, a discussion of these techniques fall outside the scope of this section. However, there is still one way left to create REST clients that we need to discuss…

Building a REST client using Visual Studio.NET 2010

Creating a REST client is made really easy in Visual Studio.NET 2010. Using Visual Studio.NET 2010, you can add a service reference to a RESTful end point. In the next procedure, you will learn how to iterate through all purchases on our test site.

Create a new C# console application via Visual Studio.NET 2010.
In the Solution Explorer window, right-click the Service References node and select Add Service Reference. This opens the Add Service Reference wizard.
Open the REST List Service in the context of your test web site. Our URL looks like this: http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc.
In the Namespace textbox, type: TestSvc.
Click OK. This adds a new service reference.

Finally, you need to add the following code to the Main() method of your console application:

string url = "http://neptune/sites/REST/TestSite/_vti_bin/listdata.svc";
TestSvc.TestSiteDataContext context = new TestSvc.TestSiteDataContext(new Uri(url));
context.Credentials = CredentialCache.DefaultCredentials;
foreach (TestSvc.PurchasesItem in context.Purchases)
{
Console.WriteLine(item.Title);
}

Alternatively, you could have opted to use Linq to interact with the Purchases result set, like so:

var purchases = from pur in context.Purchases select pur;

As you can see, interacting with a RESTful service is a breeze in Visual Studio.NET 2010.

Binary Large OBjects: Externalizing BLOB storage via RBS

vrijdag 30 april 2010 11:33:26

Binary Large OBjects: Externalizing BLOB storage via RBS

Written by: Margriet Bruggeman, Nikander Bruggeman.

April 30, 2010

Binary Large OBjects: Externalizing BLOB storage via RBS

Before the 2008 release of SQL Server, you could choose whether a custom application stored large pieces of data (Binary Large Objects, or BLOBs) inside the database or maintained references to BLOBs on the file system. The FILESTREAM feature of SQL Server 2008 changed all that by offering the best of both worlds.

The drawback of storing BLOBs in data columns (with an upper limit of 2GB) is that it typically results in lesser performance because streaming BLOB data from a database is not nearly as efficient as streaming BLOB data from the file system, since the file system is optimized for streaming unstructured data. The SQL Server 2008 FILESTREAM feature allows you to configure a database so that it looks as though BLOB data is stored in the database (inline with the table data), whereas in reality the BLOB data is stored externally on the file system. This leads to several good things: BLOBs still benefit from the features offered by the database server such as transaction support, SQL Server manages links to BLOBs on the file system automatically, and automatic inclusion in backup/restore scenario's, while at the same time also gaining the performance benefits offered by the file system.

SharePoint implementations have two interesting traits that make externalizing BLOB storage an extremely interesting topic:

SharePoint implementations contain loads of pieces of BLOBs. According to Microsoft, this typically accounts for 60-70% of the total content.
SharePoint isn't very good in handling large files. Although the usual Office files are seldomly larger than 10MB, you probably will encounter issues once you cross the 25MB border. Some files, such as media files, advanced drawings, and scanned documents, are way larger than that and that is a sure way to ask for trouble. It depends on a lot of factors if working with large files will result in issues, if you want to learn more we have explored this topic in detail in our blog post "Working with large files" which can be found at http://www.lcbridge.nl/vision/2008/largefiles.htm.

As a side note, high-end document management systems don't usually store documents in a database. Instead, they store them on the file system or in a custom repository that is optimized for handling large files and large amounts of files. At one time, we were hired by a company that builds a high-end document management system. During our stay, the company tried to build in support for WinFS, but unfortunately WinFS itself died a slow death (again), so they had to abandon that idea.

We guessed the SharePoint 2010 product team acknowledged these facts which lead to the introduction of the External BLOB Storage (EBS) feature in SharePoint 2007 sp1, which allowed developers to create custom external BLOB store providers that allowed the storage of BLOBs in the external repository of your choice.

In case you are wondering, EBS is deprecated in SharePoint 2010 and you don't need it anymore because SharePoint 2010 introduces the Remote BLOB Storage (RBS) feature, which is the actual topic of this section.

Note. Other than providing a historical context, there is a reason why we brought up the SQL Server 2008 FILESTREAM at the beginning of this section. As you will see later, you can use RBS in conjunction with the FILESTREAM feature.

The following components are of interest when it comes to understanding RBS:

The SharePoint object model interacts with SQL Server configuration and content databases.
RBS introduces a new SQL RBS Client Library that allows the SharePoint 2010 object model to store BLOB data externally.
An RBS Provider API allows storage vendors to create an external storage system for SharePoint 2010. You should note that the adoption of a provider model makes it extremely easy to switch to another provider, which ultimately comes down to choosing another repository for storing BLOBs.
An RBS installation requires one or more BLOB store providers implementing the Provider API. Please note that you can associate one active BLOB store provider for a specific content database.
Every BLOB store provider uses it's own specialized type of BLOB store. BLOB stores are only accessed through their custom providers.
RBS Maintainer. The RBS Mainainer runs as a Windows scheduled task that is responsible for handling maintenance tasks and doing garbage collection. It can either be installed on a web front-end (WFE) or on the database server itself.

Note. Currently, several vendors (EMC², Open Text, NetApp, AvePoint, and CommVault) are developing external storage systems for SharePoint 2010. Typically, you can expect that these systems will be very good at storing and managing large BLOBs, but will also have advanced features such as support for granular disaster recovery scenarios, BLOB immutability (which causes a BLOB image to stay the same, newer versions of the same document will be stored as separate BLOB images), expunging capabilities (which refers to really deleting BLOB data, or better said: obliterating data beyond recovery, for safety reasons), data de-duplication, and the possibility to enforce guaranteed retention and deletion policies. Such capabilities transcend BLOB storage to a whole new, high-end level. As a side note, you can also create your own BLOB storage system, but we guess you should never walk down that road unless you want to become a BLOB storage vendor yourself.

Architecturally, RBS looks like this:

If you enable RBS and store a document, the following happens:

The client issues a save request.
The SharePoint object model enforces business logic such as the checking of credentials, extracting metadata and so on.
The SharePoint object model calls the RBS client library on the WFE and asks it to save the BLOB.
The RBS client library locates the appropriate BLOB store provider.
The BLOB store provider stores the BLOB in it's associated BLOB store and returns a unique BLOB id. Through the RBS client library, this id is eventually returned to the SharePoint object model.
The SharePoint object model saves metadata and the BLOB id in the SharePoint content database.

As you would expect, this process is entirely transparent to the end user. At the front-end, there is no way to determine if a BLOB was stored in either a custom BLOB store or the SharePoint content database.

If you save a document and RBS is enabled, the following happens:

The client issues a request to open a document.
Again, the SharePoint object model enforces business logic such as the checking of credentials.
The SharePoint object model calls the SharePoint content dabase and retrieves the BLOB id and associated metadata. It may do so in advanced ways, such as requesting multiple BLOB id's for multiple documents at the same time (although you can't influence such implementation details).
The SharePoint object model passes the BLOB id to the RBS client library.
The RBS client library calls the appropriate BLOB store provider.
The BLOB store provider reads the BLOB data and returns it to the RBS client library, which returns it to the SharePoint object model, which returns it to the ultimate destination: the client.

RBS offers the following advantages:

Custom BLOB stores are better suited for handling BLOB data.
The most simple form of a BLOB store, the file system, is cheaper than SQL storage (such as SAN storage).
It allows you to manage document metadata and the document BLOBs themselves separately. This allows you to set up more cost effective backup/restore scenario's such as separate back up schedules (for instance, in backup/restore scenario's).
It may give you, depending on the BLOB store you choose, advanced storage capabilities. Please refer to the previous side note about BLOB store provider vendors for more information.
Better support for handling hierarchical data.

RBS is a separate downloadable component that is a part of the SQL Server 2008 R2 feature pack. In order to find it, just search for "download rbs sharepoint" in your favorite search engine to find the latest location on http://go.microsoft.com. You can also check the SQL Remote Blob Storage team blog at http://blogs.msdn.com/sqlrbs. At the time of writing, the most recent version of RBS is called RBS 2008 R2 with FILESTREAM provider, which is supported on both SQL Server 2008 and SQL Server 2008 R2. Luckily, as it's name implies, RBS is shipped with a BLOB store provider for the file system, so you don't need to purchase a 3rd party vendor BLOB store provider in order to test RBS.

Note. Do you remember that we promised that RBS was able to leverage the FILESTREAM feature of SQL Server 2008? Here it is, in order to use the BLOB store provider for the file system, you need to enable the FILESTREAM feature on SQL Server 2008. Because of this, all BLOBs are stored on the local file system of the SQL Server.

If you want to install RBS and use the file system BLOB store provider that is available out-of-the-box, you need to do the following:

Make sure you are using the Enterprise Edition of SQL Server 2008.
Enable the SQL Server 2008 FILESTREAM feature.
Alter your SharePoint content database so that it supports the FILESTREAM feature as well.
First install RBS on SQL Server.
Install RBS on every WFE.
Enable RBS using PowerShell.

Please note that you can also use PowerShell to migrate BLOBs that are currently stored in the SharePoint content database to the BLOB store of the RBS provider that is currently active for the content database.

Enabling the FILESTREAM feature on SQL Server 2008

You need to enable the FILESTREAM server for a specific SQL Server instance first. Please note: In this example we are using SQL Server 2008 R2. The following procedure explains how to do this:

Click Start > All Programs > Administrative Tools > Microsoft SQL Server 2008 R2 > Configuration Tools > SQL Server Configuration Manager. This opens Sql Server Configuration Manager.
Click on the SQL Server Services node. This opens a list of all available SQL Server services.
Right-click the instance of SQL Server (MSSQLSERVER) where you need support for the FILESTREAM feature and choose Properties. This opens the SQL Server (MSSQLSERVER) Properties window.
Click the FILESTREAM tab.
Click the Enable FILESTREAM for Transact-SQL access checkbox.
Click the Enable FILESTREAM for file I/O streaming access checkbox.
If your working on a development machine, go ahead and click the Allow remote clients to have streaming access to FILESTREAM data as well.
Click OK.

This doesn't finish the process. You still need to enable the FILESTREAM feature for the SQL Server instance. You can do this in two ways. The next procedure explains how to enable the FILESTREAM feature by calling a stored procedure.

Open SQL Server Management Studio, select the SQL Server instance you want to manage and click Connect.
Click New Query.
Issue the following command: EXEC_sp_configure filestream_access_level, 2.
Issue the following command: RECONFIGURE.

If you don't enable FILESTREAM features often, you are probably better of using the next method, as it's easier to remember in the long run and makes it more clear to see what you are doing. The next procedure explains how to enable the FILESTREAM feature via the user interface:

Open SQL Server Management Studio, select the SQL Server instance you want to manage and click Connect.
Right-click the [name of SQL Server instance] node and choose Properties.
Click Advanced.
In the Filestream section, set the FileStream Access Level drop down list to Full access enabled.
Click OK. Aninformational message appears stating that you need to restart SQL Server if you want your configuration changes to take effect.
Click OK.
Right-click the SQL Server node and choose Restart.

Enable FILESTREAM for your SharePoint 2010 content database

In this section, we will discuss how to enable FILESTREAM support for your SharePoint 2010 content database. First., you will need to determine the name of the SharePoint 2010 content database associated to your site collection.

Please note: In this example, we're expecting that your site collection is only associated to a single content database. In more advanced scenarios, this may not be so. In such a case, you need to iterate through the entire collection of content databases and locate the ones you want to change. At a later stage this issue pops up again, because soon you will be enabling RBS per content database via PowerShell scripts.

Click Start > All Programs > Microsoft SharePoint 2010 Products > SharePoint 2010 Management Shell. This opens the Administrator: SharePoint 2010 Management Shell.
Issue the following command: $site = get-spsite "http://[URL of site collection]", for instance $site = get-spsite "http://sharepoint2010.
Issue the following command: $site.ContentDatabase.

Take a note of the name of your SharePoint content database. The name of ours is WSS_Content, and we'll use this name throughout the example. Then, you need to open up a query window in SQL Server Management Studio and issue three queries. First, you need to create a unique key.

use [WSS_Content]
if not exists (select * from sys.symmetric_keys where name = N'##MS_DatabaseMasterKey##')create master key encryption by password = N'[enter a password here]'

Then, you need to create a new file group for the FILESTREAM feature by entering the following command:

use [WSS_Content]
if not exists (select groupname from sysfilegroups where groupname=N'RBSFilestreamProvider')alter database [WSS_Content]
add filegroup RBSFilestreamProvider contains filestream

Finally, you need to add a new FILESTREAM file to the file group by issuing the following command:

use [WSS_Content]
alter database [WSS_Content] add file (name = RBSFilestreamFile, filename = 'c:\Blobstore') to filegroup RBSFilestreamProvider

This causes BLOB files to be stored on the following location of SQL Server: c:\Blobstore.

Note. You can copy the commands from the TechNet article "Install and configure Remote BLOB Storage (SharePoint 2010).

Install RBS on SQL Server

You can't directly click on RBS_X64.msi and follow the instructions of the installation wizard because the wizard sets default values that are not recommended for SharePoint 2010. Instead, do the following:

Click Start and locate the Command Prompt icon.
Right-click the Command Prompt icon and choose Run as administrator. This opens the Administrator: Command Prompt window.
Issue the following command (and replace the values for both DBNAME and DBINSTANCE with the names of your SharePoint content database and SQL Server instance name respectively): msiexec /qn /lvx* rbs_install_log.txt /i RBS_X64.msi DBNAME="WSS_Content" DBINSTANCE="DBInstanceName" ADDLOCAL="Client,Docs,Maintainer,ServerScript,FilestreamClient,FilestreamServer"

Although the command prompt returns immediately, the msi file is installed in quiet mode. If you want to be informed explicictly, change the /qn parameter to /qn+, otherwise, you need to wait a couple of minutes and check if the msiexec process is still running via Task Manager.

After installation, the folder containing the RBS binaries should contain a file called rbs_install_log.txt. The last 20 lines should include the message that "Installation completed successfully".

Install RBS on WFEs

After installing RBS on SQL Server, you will also need to install RBS on all WFEs. You can do this by issung the following command from the Administrator Command Prompt window:

msiexec /qn /lvx* rbs_install_log.txt /i RBS_X64.msi DBNAME="WSS_Content" DBINSTANCE="DBInstanceName" ADDLOCAL="Client,Docs,Maintainer,ServerScript,FilestreamClient,FilestreamServer"

After installation, the folder containing the RBS binaries should contain a file called rbs_install_log.txt. The last 20 lines should include the message that "Installation completed successfully". You may also note that the SharePoint content database now contains several new tables that are prefixed with "mssqlrbs". You can check this using the following SQL command from SQL Server Management Studio within the context of the SharePoint content database:

select * from dbo.sysobjects where name like 'mssqlrbs%'

PowerShell time: enable a provider for a content database

Choose any given WFE within the SharePoint farm and follow the next procedure to enable the file system BLOB store provider that is available by default for a given content database.

Click Start > All Programs > Microsoft SharePoint 2010 Products > SharePoint 2010 Management Shell. This opens the Administrator: SharePoint 2010 Management Shell.
Issue the following command: $cdb = Get-SPContentDatabase –WebApplication http:/[name of site collection]
Alternatively, if your site collection is associated to multiple content databases, issue the following command: $cdb = Get-SPContentDatabase –WebApplication http://sitename
Then, issue the following commands to finish the configuration: $rbss = $cdb.RemoteBlobStorageSettings
$rbss.Enable()
$rbss.SetActiveProviderName($rbss.GetProviderNames()[0])
$rbss

Note. If you need more information for working with PowerShell, and working with collections in PowerShell in particular, please refer to section "PowerShell: Retrieve information about SharePoint in the fastest way possible".

At this point, the file system BLOB store provider is enabled for your SharePoint 2010 content database. There is one quite cool feature that you still need to consider. You can set the minimum BLOB storage size. This allows you to determine that all files smaller than the specified threshold are still stored in the SharePoint content database itself, but all files larger than the threshold will be stored in the BLOB store. You can do this by issuing the following commands via PowerShell:

$cdb = Get-SPContentDatabase –WebApplication http:/[name of site collection]
$cdb.RemoteBlobStorageSettings.MinimumBlobStorageSize = [number of bytes]
#cdb.Update()

If you start adding files to SharePoint that surpass this threshold, in the user interface you will notice exactly… nothing! If you want to check that the document was indeed added to the BLOB store you can open SQL Server Management Studio, open the SharePoint content database and issue the following command from the query window:

SELECT siteid, content, rbsid FROM alldocstreams WHERE rbsid IS NOT NULL

Normally, the content column contains the BLOB. For documents stored in the BLOB store, this column is NULL and the column rbsid will contain a BLOB id that the BLOB provider uses to locate the correct BLBO file. So, if the previous queries returns results, you know that you have successfully added a file to the BLOB store. Alternatively, you could also check the contents of c:\Blobstore and see if it contains any entries.

Concluding the RBS section

To conclude the discussion of RBS. In general, RBS has one minor disadvantage: it makes a solution more complex to manage, because you need to maintain two stores (SQL Server and the BLOB store) instead of one. This particularly becomes obvious in back-up/restore scenario's. In complex scenario's where you really do need the benefits offered by RBS, we don't think this should be a factor in your decisioning making.

Other possible disadvantages are completely dependent upon the specific BLOB store provider you are considering. The file system BLOB store provider that is shipped out-of-the box is only able to store BLOBs on the local file system of the SQL Server instance containing the SharePoint 2010 content dabase. This makes it unsuitable for larger deployments. We don't have a lot of performance data about the file system provider yet, but it wouldn't surprise us if it needs to mature first in order to really become useful in production environments. As it stands now, we would definitely consider using the file system provider in intranet scenario's where a small number of users working in locations close to each other are working with large files (> 25 MB) on a regular basis. In other scenario's, probably not, although we're the first ones to admit that we haven't collected enough data to back this statement up.

As for the benefits: they depend completely on the choice of BLOB store provider too, although you can safely expect that this is the part where RBS really shines. Once vendors start delivering features such as performance optimizations for large BLOBs, expunging capabilities, and retention policies, RBS can lift SharePoint 2010 to a whole new dimension. We bet that RBS will become a huge factor in the high-end market for SharePoint 2010. Other implementations can safely ignore it and live happily ever after storing BLOBs in the SharePoint 2010 content database.

Enterprise Search in SharePoint 2010: brief discussion

vrijdag 30 april 2010 11:33:26

Enterprise Search in SharePoint 2010: brief discussion

Written by: Margriet Bruggeman, Nikander Bruggeman.

April 30, 2010

Enterprise Search in SharePoint 2010: brief discussion

Although the biggest change in the SharePoint search arena, as far as we're concerned, consists of the addition of Fast Search to SharePoint (see section "Fast Search Server 2010 for SharePoint"), there have been additions to the existing enterprise search features of MS Search as well. In this section, we will quickly touch on them.

You can now perform fuzzy name searches. If you search for "Nik", you may find "Nick", and "Nico" as well. Search center now allows you to use the Boolean operators AND, OR, and NOT. Search queries support wildcards at the end of a search term, for instance, you can search for "la*" and find results such as "large" and "laugh". You can also use Windows Explorer to search for documents in SharePoint sites. Search results can be navigated via a refinement panel (also known as drill-down or faceted search), that allow you to refine search results based on categories such as author, data, and document type. Search result summaries have been improved and include additional information such as contact persons, rating, and document tags. Search Center provides suggestions while you type a query. Site administrators can manage a central list of taxonomy keywords that are assigned to a specific column. Document owners can take these predefined and pre-associated taxonomy keywords and use them when definining document metadata, which makes a document easier to find. Searching for your own name yields additional information, such as the number of times your profile has been viewed in a given period, or keywords that have been used to find you. Probabalistic ranking has been improved in several ways: the click-through history has become an important ranking factor, if a specific document has been visited more than others it's ranking is promoted. Inferred metadata influences ranking as well, if a document is missing certain pieces of metadata (such as: it's author), the crawling process will try to infer the missing metadata in an intelligent way from the contents of a document. Finally, social distance has become an important ranking factor in people search. Search results of graphics in Image libraries now also include metadata such as it's size and last modification date. The current document indexing limit lies around 50 million documents. Enterprise Search is shipped with a connector for crawling Exchange 2010 public folders, and there is a framework for creating custom connectors too. Finally, topology management has improved as well. In SharePoint 2010, it is easy to increase the number of query servers and/or index servers. You can also configure a primary index server and a secondary, redundant index server to avoid system down time.

Access Services: Creating web applications and publishing them to SharePoint 2010

donderdag 29 april 2010 12:23:29

Access Services: Creating web applications and publishing them to SharePoint 2010

Written by: Margriet Bruggeman, Nikander Bruggeman.

April 29, 2010

Access Services: Creating web applications and publishing them to SharePoint 2010

Every now and then a technology appears that promises that end users will be able to create CRUD (Create Read Update & Delete) applications in a point-and-click fashion. In the previous two versions of SharePoint, this role was acted out by SharePoint Designer. In the 2010 release, this role is fulfilled by Access 2010 in combination with the newly introduced Access services.

You may have worked with InfoPath and Forms Services before. InfoPath allows you to create a form, and as long as it's web compatible you can publish it, via Forms Services, to SharePoint. Once an end user opens such a form, Forms Services knows how to render it to HTML and JavaScript. Access Services is much like that. Access 2010 allows you to create a web application and publish it, via Access Services, to SharePoint 2010. Since we feel the greatest strength of Access is, and has been for years, the ease with which you can create CRUD applications we think it is worth your time taking a look at how to integrate Access 2010 web applications with SharePoint 2010.

You can either use a browser or Access as the client to communicate with Access publications that have been published through Access services (from now on, we will call such applications: Access web applications). Both clients communicate with Access Services, although a browser leverages the data form web part (which renders Access forms and reports) and the Project datasheet object (which renders Access datasheets) to do so, whereas the Access client directly talks to Access Services. In the middle tier, an Access services web service provides query processing and data access logic, which mainly have to do with caching of data and performance optimization. Ultimately, this middle tier is responsible for managing all application data, which is stored in SharePoint lists. Any advanced forms of business logic you may have implemented in Access via macros will be implemented by Access Services with the help of list event handlers, SharePoint workflows and QuickFlows.

In the rest of this blog post, we will create a small Access web application that contains a Products table and a Purchases table. We will create the required database tables for this application, create an overview/detail forms for it, and publish it to SharePoint 2010 using Access Services. In the following procedure, we will create a new database and a new table that will hold Product information.

Click Start > All Programs > Microsoft Office > Microsoft Access 2010.
On the File tab, ensure that the New section is highlighted and choose Blank Web Database.
In the File Name textbox (located in the Blank database pane), specify the following value: ProductDatabase.accdb.
Click the Create button. This causes the creation of a new database and allows you to create a new table.
In the Table Designer window, click the Click to Add column header and choose Text.
Rename Field1 to Product.
Add another Text column and call it: Description.
Add a new Date & Time column and call it: Available.
Click the Save button and name the table Products.

You have created a simple products table. Provide some test data for it and save the data. Our Products table looks like this:

Next, add a new table that will contain purchases of these products.

Click the Create tab and choose Table. This opens a new table.
Add a new column of type Number and rename it to: Item_Total.
Add a new column of type Currency and rename it to: Price.
Add a new column of type Lookup & Relationship. This opens the Lookup wizard.
Make sure the option I want the lookup field to get the values from another table or query is selected and click Next.
Select the Products table. Click Next.
Select the ID field and click the > button.
Select the Product field and click the > button.
Click Next.
Specify the Product column in the Sort order drop down list. This orders items according to their product name.
Click Next.
Make sure the Hide key column checkbox is selected. Accept all other default values and click Next.
Now, you are asked to provide a label for your lookup column. Just call it: Product.
Click the Enable Data Integrity checkbox and accept the Restrict Delete option which prevents Products from being deleted accidentally.
Click Finish. This opens the Save As dialog window.
In the Table Name textbox, enter the value: Purchases. Click OK.

Next, we will add a macro that notifies a sales manager every time a new purchase takes place.

Open the Purchases table by double-clicking it's name in the Tables section.
Click the Table tab.
Click the After Insert button. This opens the Macro Tools designer.
Select the SendEmail action and specify a To address, a Subject, and a Body text.
Click the Save button, followed by the Close button (located at the upper right hand corner).
The next step is to create an Overview page of all current purchases.
Double-click the Purchases table in the Tables section of the Navigation pane. This opens the Purchases table.
Click the Create tab.
Click Datasheet. This opens a Datasheet window.
Save the Overview form by pressing CTRL+S. A Save As dialog window appears.
Enter the following Form Name: PurchaseOverview.
Click OK.

You should notice that the Navigation pane now holds a new section called Forms, which in its turn contains the PurchaseOverview form you have just created. Now we will also create a purchase details form.

Again, click the Purchases table.
Click the Create tab, then click the Form button. This creates a new form called Purchases, containing all fields in the table.
Click the Save button and name the form: PurchaseDetail. Click OK.

Now you should have two forms. In the next part we will create a navigation form.

Again, click the Purchases table.
Click the Create tab, then click Navigation > Horizontal tabs. This opens a new Navigation form.
Drag-n-drop the PurchaseOverview form on the Navigation form.
Double-click the PurchaseOverview tab and rename it to Overview of purchases. The Navigation form should now look similar to this:
Save the form and call it: Home, then click OK.

In the next step, we will enhance the Home Navigation form so that you can open a detail form from the overview form.

Right-click the Home Navigation form and choose Layout View. This opens the Layout View of the form.
Select the ID column.
Press F4 to open the Property Sheet of this column.
Click the Event tab.
Click the On Dbl Click event, then click the … (Property Builder) button. This opens the Choose Builder dialog window.
Select the Macro Builder and click OK.
Select the If action.
In the If condition text box, add the following value: Not IsNull([ID]).
In the Add New Action drop down list, choose OpenForm.
In the Form Name drop down list, select the PurchaseDetail form.
In the Where Condition text box, enter the value: [ID] = [Forms]![Home]![NavigationSubForm].[Form]![ID]. Don't worry, IntelliSense will help you accomplish this.
In the Data Mode drop down list, select Edit.
Set the Window Mode drop down list to Dialog.

At this point, the Macro should look like this:

Go ahead and save and close the Macro. You can reopen it any time via the Event tab of the Property sheet. Next, you will need to specify a default form that opens once users start interacting with the Products database.

Click the File tab, then click Options. This opens the Access Options dialog window.
Select the Current Database option.
In the Display Form drop down list, select Home.
Click OK. An informational message appears stating that you must close and reopen the current database for the specified option to take effect.
Click OK.

Close Access 2010 and re-open it, then re-open the ProductDatabase database. This time, the Home Navigation Form should be selected by default. Now it's time to publish the application to SharePoint 2010. First of all, you should have a SharePoint site that you want to publish the Access application to. In our example, we will use a site called AccessHost located on a SharePoint site collection at http://sharepoint2010.

Click the File tab, then click the Share option.
Locate the Share section and click Publish to Access Services.
Click the Run Compatibility Checker button. This may ask you to close any objects you still have open. The main thing this button does is check to see if your application is web compatible (in other words: is supported in Access services).
Locate the Publish to Access Services section.
In the Server URL text box, add the value: http://sharepoint2010.
In the Site Name text box, add the value: MyAccessApp.
Click the Publish to Access Services button.

The publication process shouldn't take long and result in the following success message:

Click on the URL that is displayed in the Publish Access Application dialog window to open your custom Access application, that is now published to Access services and can now be accessed as a normal web application. First, it will open the default navigation form you have created:

You can double click on the ID field to open the Detail form: Access Services has taken the Macro we've created earlier and translated it to JavaScript. The Detail form itself looks like this:

If at one time you feel the need to change your web application, you can do this by opening your database, click on the Info tab, and then click the Sync All button.

In the final part of this section, we will discuss how to deploy Access web applications. In essence, the steps for creating an Access web application remain the same, only you won't publish the Access application directly to Access services. Instead, all you have to do is create a template of your Access application, upload it as a user solution. Once you have done that, you can create new SharePoint sites that are based on this template to create new instances of your Access web application. The next procedure explains how to deploy an Access web application via a template.

Open your Access application.
Click the File tab.
Click Share.
In the File Types section, click Save Database As.
In the Save Database As section, under Database File Types, click Template. This is shown in Figure x. This opens the Create New Template from This Database dialog window.
In our case, we will create a new template and call it Purchases Web App Template, and choose to include the current data in the template.
Click OK. You should see an informational message telling you that the template file (*.accdt file) is save locally.
Now, open the SharePoint 2010 site collection where you want to add the Access web application template.
In the root site of the SharePoint 2010 site collection, click Site Actions > Site Settings. This opens the Site Settings page of the root site of the SharePoint 2010 site collection.
In the Galleries section, click Solutions. This opens the site collection Solutions page.
Click the Solutions tab.
Click the Upload Solution button. This opens the Upload Document dialog window.
Browse to your template file, double-click it, then click OK. This adds the solution (a normal SharePoint *.wsp file) to your gallery.
Then, select the template file, choose the item action menu and click Activate. This opens the Solution Gallery - Activate Solution dialog window.
Click the Activate button.
Go back to the site collection and choose Site Actions > New Site. This opens the Create page.
Among the options listed on the Create page, you should see your own custom template. If you select it and create a new site, you will have successfully created a new instance of your Access web application in the form of a new SharePoint site.

As you can see, the concept of Access templates that can be exported as user solutions make it extremely easy to deploy and reuse Access web applications.

Velocity: Solving a Spider-Man/Human Torch dilemma

04 May 2009 10:09:13

Velocity: Solving a Spider-Man/Human Torch dilemma

Written by: Margriet Bruggeman, Nikander Bruggeman.

Velocity: Solving a Spider-Man/Human Torch dilemma

It may come as no surprise that people who call their own consulting company Lois & Clark IT Services like to read comics about super heroes occasionally. One of our favourites is an old one where Spider-Man and the Human Torch team up to fight the Sandman. They end up losing badly and Sandman takes them to an old, deserted silo that’s filled with water. In the story it’s Christmas eve, which motivates Sandman to come up with an escape route for both super heroes and he tells Spider-Man and the Human Torch that they will be able to save their lives if they’re smart enough to figure out this escape route. Then, he ties them up with some kind of heavy rope in such a way (with their hands behind their backs and hands tied to each other) that only one hero at a time is able to breath air, and the other hero is held under water and needs to hold his breath. After swapping places with the Human Torch for some time, Spider-Man realizes they can’t go on like that forever and in a moment of enlightenment sees the escape route. He swaps places with the Human Torch, submerges under water and stays there for as long as he can (and maybe even a bit longer than that). Finally, the Human Torch catches on when he realizes his face has dried up enough to light part of his flames, which allows him to burn through the ropes and free both of them. We fondly call this little story the Spider-Man/Human Torch dilemma and use it when referring to situations where multiple parties are involved and one of them has to suffer to achieve a greater, common goal. In this article we’ll explore a Spider-Man/Human Torch dilemma that revolves around MOSS performance and memory management.

Train of thought #1: What do we want to achieve?

The premise of the Spider-Man/Human Torch is that there is a greater common goal that you want to achieve. Let’s start the explanation of the particular problem we’re writing about with the statement that (in our minds) MOSS has never been a fast product. It’s first version, SharePoint Portal Server 2001 wasn’t brilliantly fast, and it’s latest incarnation, MOSS 2007, still isn’t exactly operating at the speed of light (although it’s true you can get it to go pretty fast).

The problem that we wanted to solve was that we wanted to create an implementation of a treeview explorer that was able to browse through various site collections, sites, lists, folders and items. This treeview explorer should respect user permissions (so it shouldn’t include items users are not entitled to see), it should show a ‘+’ sign if (and only if) items contain sub items (e.g. a folder that contains zero documents or folders isn’t worth one of the end users precious mouse clicks). We also wanted to group folders and group documents and list them alphabetically. The treeview explorer should be able to retrieve information via client callbacks, so the entire page doesn’t need to be refreshed and it should be able to exclude items based on either application configuration information or item metadata. Most of all, this treeview explorer should be blazingly fast. The next figure shows an example of the way this treeview explorer should work:

Train of thought #2: What methods are available for achieving our goal?

When beginning this journey, we thought it would be logical to take a close look at the available methods that could be used when building such a solution and see if there is one that is “blazingly fast”. The white paper “White paper: Working with large lists in Office SharePoint Server 2007” is a nice starting place to explore available methods for working with list and folder contents (which eventually turned out to be the main pain point during implementation of this particular treeview).

According to the test results described in this white paper, when working with large lists and folders, the order (based on performance) of available methods is:

PortalSiteMapProvider (provides blazing fast performance)
SPList with SPQuery (fast)
SPListItem with DataTable (fast)
Search (fast)
Lists web service (fast)
SPList with DataTable (extremely slow)
SPList with For/Each (extremely slow)

By default, we decided to dismiss a couple of the approaches mentioned above:

The Lists web service, in our minds it was impossible that this would turn out to be the fastest method for accessing list and folder content because of the communication overhead.
Search, this approach has some disadvantages (such as the fact that content first needs to be indexed and that the treeview needs to interact with a set of list or folder items to determine if there are child items) that made us forget about this method too.
SPList with DataTable, according the aforementioned white paper this was an extremely slow method, so we didn’t feel the need to go out and give this method a go.

Before actually trying it out, we were pretty excited about the PortalSiteMapProvider approach. It turned out to be pretty limited in functionality (in particular when it comes to ways to support advanced caching scenarios) and especially useless in our situation because we’re frequently retrieving different data sets. As a result, we had to discard this approach as well. By the way, we have read several blog posts that are really enthousiastic about this approach, although unfortunately some authors certainly give the impression that they’ve just read about it without actually trying it.

There was another option that was hard to resist: directly querying the content database. We knew this approach would most likely give us the performance boost we were searching for. The topic of accessing SharePoint databases directly has been debated quite often, most people rejecting this solution vehemently. The discussion basically revolves round the problem that it’s not supported by MS, although nowadays the stored procedures are documented to some level and are called the “Database End-User Experience Communications Protocol” (and according to http://msdn.microsoft.com/en-us/library/cc341380.aspx it’s ok for third-party components to use those).

By the way, we always wonder if the existence of this protocol is a side-effect of the anti-trust lawsuits against Microsoft instigated by Neelie Kroes or if there is an actual reason this “protocol” documentation is released and that people are actually using it. If you have used this protocol to your benefit in a scenario that’s supported by Microsoft, we sure would love to hear more about it!

Let’s get back to the last topic, the topic of directly querying the content database. We’ve established that most people see this as a no-no. On the other hand, if you make sure to avoid read locks, such a solution should be okay, as long as you accept the fact that hot fixes and service packs might break the code. The main reason why we’ve decided not to pursue the approach of directly accessing the content database is because a solution accessing SharePoint databases directly won’t meet the requirements of most SharePoint code acceptance checklists out there, thereby pretty much annihilating the usefulness of our treeview explorer. The current state of the “Database End-User Experience Communications Protocol” is that it is too poorly documented and too incomplete as a choice we’re willing to rely on, so we had to drop that option too.

Train of thought #3: Are the aforementioned methods adequate?

After establishing which methods were available to us, we needed to discern which ones (if any) were suitable for our purposes. The next step was that we’ve created a little test tool that’s able to generate simple folder and item structures. We made it create a folder containing 1000 subfolders and 1000 documents. That makes a total of 2000 list items, the magic number when it comes to SharePoint performance limits.

We decided to pursue our exploration by investigating the following methods:

SPList with For loop
SPList with For/Each
SPList with SPQuery

SPList with For loop

The first available path that we wanted to check out was to count the number of items in a folder, loop through every sub item, and add it to our treeview. We’re displaying a simplified version of this approach in the next code example, in which we’ll also check if a subfolder contains child items:

// folder contains a valid reference to an SPFolder object.
// We haven’t included the code prior to this point.
SPFolder folder = web.GetFolder(strFolderPath);
int countFiles = folder.Files.Count;
for ( int i = 0; i < countFiles; i++)
{
  sb.Append(folder.Files[i].Name + "\r\n");
}

int countFolders = folder.SubFolders.Count;
for (int j = 0; j < countFolders; j++)
{
  sb.Append(folder.SubFolders[j].Name);
  sb.Append("\r\n");
}

The end result of this test wasn’t bad at all. The test code looped through a total of 2000 child items. Test results fell in the 900 msec – 1100 msec range, which was better than we expected.

SPList with For/Each

The next available path was to loop through each child item of a folder and add the results to our treeview. We’re displaying a simplified version of this approach in the next code example, in which we’ll also check if a subfolder contains child items:

// folder contains a valid reference to an SPFolder object.
// We haven’t included the code prior to this point.
SPFolder folder = web.GetFolder(strFolderPath);
foreach (SPFile file in folder.Files)
{
  sb.Append(file.Name + "\r\n");
}

foreach (SPFolder subFolder in folder.SubFolders)
{
  sb.Append(subFolder.Name);
  sb.Append(" subfolders: "
  + subFolder.Files.Count
  + " sub items: "
  + subFolder.SubFolders.Count);
  sb.Append("\r\n");
}

The test code looped through 2000 child items. Test results fell in the 3300 msec – 4500 msec range, which wasn’t a result we felt particularly cheerful about.

Please note: There’s an important difference between this code fragment and the code fragment discussed in section “SPList with For loop”. In this code fragment we’re also retrieving the number of files and subfolders in every folder, which accounts for the performance difference between both approaches. In section “Why the performance difference between different iteration methods?” we’ll discuss this difference in detail.

SPList with SPQuery

Based on other tests we’ve seen, interacting with a list using SPQuery performs pretty well. Because of this, we were very interested in the results. The following code is a simplified version of the code we’ve used to measure the performance of this approach:

SPList list = web.Lists[List];
SPFolder folder = web.GetFolder(“[folder path]”);

SPQuery query = new SPQuery();
query.Folder = folder;
query.ViewFields = "";
query.Query = "";

SPListItemCollection items = list.GetItems(query);
foreach (SPListItem item in items)
{
  if (item.FileSystemObjectType == SPFileSystemObjectType.Folder)
  {
    sbFolder.Append("Folder: " + item.Name);
    sbFolder.Append("\r\n");
  }
  else if (item.FileSystemObjectType == SPFileSystemObjectType.File)
  {
    sbFile.Append("File: " + item.Name + "\r\n");
  }
}

sb.Append(sbFolder);
sb.Append(sbFile);

The results are good, test results fell in the 900 msec – 1200 msec range. This was very comparable to the results found in the For loop test, although in our measurements the SPQuery code was a tad slower. Please note, the SPQuery approach still requires us to interact with subfolders because we need to determine if a subfolder has children (which affects the way our treeview renders the user interface).

Why the performance difference between different iteration methods?

There is a huge difference in the performance of different iteration methods. Why is that? After running a trace in SQL Server Profiler things become much clearer. Let’s take a closer look at the results.

For loop

The sample code acceptance checklist created by MS (http://technet.microsoft.com/en-us/library/cc707802.aspx) warns us not to use the Count property of an SPListItemCollection in a loop, but to store it’s value in a variable instead. In this and the next section we’ll explore if that also holds true when calling the Count property of an SPFolder object. In the next example, we’re calling this Count property repeatedly:

int countFiles = folder.Files.Count;
for ( int i = 0; i < folder.Files.Count; i++)
{
  sb.Append(folder.Files[i].Name + "\r\n");
}

int countFolders = folder.SubFolders.Count;
for (int j = 0; j < folder.SubFolders.Count; j++)
{
  sb.Append(folder.SubFolders[j].Name);
  sb.Append("\r\n");
}

When running the SQL Server Profiler tool, we’ve noticed that the execution of this code results in two stored procedures being fired, called proc_GetTpWebMetaDataAndListMetaData* which provides metadata about a site or list (http://msdn.microsoft.com/en-us/library/cc668944(PROT.10).aspx) and proc_ListUrls which, amongst other things, returns list items (http://msdn.microsoft.com/en-us/library/cc669002.aspx).

* This particular stored procedure doesn’t seem to be a part of the protocol documentation mentioned in “Train of thought #2” (although we may have overlooked it) so we’re concluding that it’s not supported to call it directly. And, in case you were keeping score, that’s the second-to-last time we’ll mention the “Database End-User Experience Communications Protocol” in this article.

For loop with storing count property

The following code is a slight variation on the code shown in the previous section “For loop”. In this example, we’re storing the Count property of an SPFolder object in a variable:

SPFolder folder = web.GetFolder(strFolderPath);
int countFiles = folder.Files.Count;
for (int i = 0; i < countFiles; i++)
{
  sb.Append(folder.Files[i].Name + "\r\n");
}

int countFolders = folder.SubFolders.Count;
for (int j = 0; j < countFolders; j++)
{
  sb.Append(folder.SubFolders[j].Name);
  sb.Append(" sub folders: " + folder.SubFolders.Count + " sub items : " + folder.Files.Count);
  sb.Append("\r\n");
}

The end result is exactly the same. Execution of this code results in the same two stored procedures being fired.

For/Each loop

The next code example discusses the For/Each loop which took a considerable longer time to execute. It’s interesting to see why that is. After executing the following code the answer was easy:

SPFolder folder = web.GetFolder(strFolderPath);
foreach (SPFile file in folder.Files)
{
  sb.Append(file.Name + "\r\n");
}

foreach (SPFolder subFolder in folder.SubFolders)
{
  sb.Append(subFolder.Name);
  sb.Append(" subfolders: "
  + subFolder.Files.Count
  + " sub items: "
  + subFolder.SubFolders.Count);
  sb.Append("\r\n");
}

This code retrieves the number of files and folders of every subfolder. First, when executing this code, the stored procedure proc_GetTpWebMetaDataAndListMetaData is called, and after that the stored procedure proc_ListUrls is called every time for every immediate subfolder in a list. So, if we were using a For/Each loop to iterate through a folder that contains 1000 subfolders (and, to make the point clearer, place 10 subfolders in every one of those subfolders), proc_ListUrls gets called 1000 times. This behaviour is not very cool and knowing this, it’s no surprise that this approach performs far worse compared to the For loop approach. Actually, considering the huge amount of work being performed in this approach, you could argue that the performance of the For/Each loop is not that bad after all.

By the way, when omitting the call to the Count property of the Files and SubFolders collections, the For/Each loop behaves exactly like the For loop. Vice versa, the For loop behaves exactly like this code fragment when calling Count properties within the loop. Therefore, using either the For loop or For/Each loop shouldn’t matter much performance-wise, a reassuring thought. A less reassuring thought is that establishing if a folder has sub items is a very expensive operation. Since we did have to know this information to successfully implement our custom treeview, the future suddenly began to look dark and grim. Using the available approaches we were bound to create a custom treeview that would amaze our end users: by being amazingly slow!

Figuring out what Browser interaction does

At this point, we were getting interested to learn what SharePoint uses itself when browsing through lists or generating treeviews.

Under water, browsing through folders and documents via a web browser and using the SPList with SPQuery method turns out to be identical. Both approaches dynamically generate a complex query and allow you to specify exactly how much results you want to retrieve (e.g. the user interface only displays 100 items at a time). The next code fragment shows a part of the dynamically generated query, which gets quite lengthy and complex:

exec sp_executesql N' SELECT TOP 101 t1.[Type] AS c0,t2.[tp_ID] AS c5c7,t1.[IsCheckoutToLocal] AS c11,UserData.[tp_ContentTypeId],t3.[nvarchar1] AS c13c6,UserData.[ntext2],UserData.[nvarchar10],UserData.[datetime1],t1.[ScopeId] AS c4,UserData.[nvarchar4],UserData.[tp_CheckoutUserId],t4.[Status1] AS c14c15,UserData.[tp_ModerationStatus],UserData.[tp_Level],t2.[nvarchar1] AS c5c6,UserData.[tp_HasCopyDestinations],t3.[tp_Created] AS c13c10,UserData.[nvarchar12],CASE WHEN DATALENGTH(t1.DirName) = 0 THEN t1.LeafName WHEN DATALENGTH(t1.LeafName) = 0 THEN t1.DirName ELSE t1.DirName + N''/'' + t1.LeafName END AS c3,t3.[nvarchar5] AS c13c9,t1.[TimeCreated] AS c2,UserData.[nvarchar3],t2.[nvarchar5] AS ed] AS c5c10,UserData.[nvarchar5],UserData.[tp_CopySource],t3.[nvarchar4] AS c13c8,UserData.[nvarchar11],UserData.[ntext3],t1.[LeafName] AS c1,t2.[nvarchar4] AS c5c8,t1.[DirName] AS c12,UserData.[tp_Editor],t3.[tp_ID] AS c13c7,UserData.[tp_ContentType] FROM UserData INNER MERGE JOIN Docs AS t1 WITH(NOLOCK) ON ( 1 = 1 AND UserData.[tp_RowOrdinal] = 0 AND t1.SiteId = UserData.tp_SiteId AND t1.SiteId = @L2 AND t1.DirName = UserData.tp_DirName AND t1.LeafName = UserData.tp_LeafName AND t1.Level = UserData.tp_Level AND (UserData.tp_Level = 255 AND t1.LTCheckoutUserId =@IU OR (UserData.tp_Level = 1 AND (UserData.tp_DraftOwnerId IS NULL OR (UserData.tp_DraftOwnerId <>@IU AND 1=0 )) OR UserData.tp_Level = 2 AND and so on… and so on…

This approach lends itself well in scenario’s where you need to apply advanced filtering during query execution. In this particular instance, we found that because we wanted to display each and every item in a list or folder we don’t get the benefits of it and don’t gain any performance boosts. Therefore, this approach wasn’t the answer either.

Then we started noticing that the out-of-the-box treeview was pretty fast (although being too limited in functionality for our tastes). Enabling the treeview in the user interface (Site Actions > Site Settings > Modify All Site Settings > [Look and Feel section] Tree view > Enable Tree View ) results in the execution of the stored procedure proc_GetListAndChildrenNSInfo which results in the retrieval of child folders (you can find a bit more information in the Content Database End-User Experience Communications Protocol Specification at http://download.microsoft.com/download/8/5/8/858F2155-D48D-4C68-9205-29460FD7698F/%5BMS-WSSEUX%5D.PDF).

This approach performs pretty well, and offers an important advantage over the For and For/Each loops because this stored procedure returns the FolderChildCounts number as well. This allows us to establish if a folder contains children without having to issue a separate stored procedure call for every subfolder. Because of this, it performs way better than the other approaches but this stored procedure still doesn’t quite suit our purposes because there are situations where we want to show documents in lists and folders as well and this stored procedure only returns folders.

So assuming you would want to call this method through the object model (like the out-of-the-box treeview does), you’d have to wonder if that is even possible. We’ve tried if we could get that to work and found that the default.master page contains an instance of an SPTreeView control which renders the treeview in the user interface. This control uses an instance of the SPHierarchyDataSourceControl class to retrieve a hierarchical view of SharePoint sites, lists, and folders (excluding sub items that are documents). This control gets it’s data from an instance of the SPHierarchyDataView class which ultimately uses the SPRequest class (located in the Microsoft.SharePoint.Library namespace). The SPRequest instance calls the all-encompassing and unmanaged owssvr.dll COM object (located in the ISAPI folder of the 12 hive) that’s responsible for communicating with the database server.

The problem is that most of the important parts of this little trail have internal access modifiers, meaning we can’t call them directly in our own code. So it certainly seems like calling the proc_GetListAndChildrenNSInfo stored procedure through the object model is out of the question (if you happen to know how to do this, please feel free to contact us at info@lcbridge.nl , excluding an approach where you need to resort to reflection to get things up and running).

On the other hand, if we’ve interpreted the MSDN statement correctly, it would be supported to call the stored procedure proc_GetListAndChildrenNSInfo directly so that would be the only option left. If only this stored procedure would have returned files as well…

Train of thought #4: Temporary Conclusion

Iterating over 2000 items in a list or folder performs pretty good. Using a For loop or For/Each translates to the same queries being executed under water. These approaches perform just as good as using the SPQuery approach.

It’s a normal request to show prefix folder icons in a treeview with a plus (+) sign and skip the plus sign if a folder doesn’t contain sub items. Implementing this wish really slows down the treeview solution and leads to an explosion of queries. To make matters worse, the out-of-the-box SharePoint treeview only displays folders and uses an internal part of the SharePoint object model to eventually call the stored procedure proc_GetListAndChildrenNSInfo which retrieves folders and folder child count in one go. Because of that, the out-of-the-box treeview indicates if a folder contains sub items and performs significantly better than a custom implementation of the treeview that does the same but uses the public part of the SharePoint object model.

Our temporary conclusion is that none of the approaches available in the SharePoint object model are very suitable for building a treeview that fulfils all the requirements described in section “Train of thought #1: What do we want to achieve?”. That is, if you want to build a treeview that is actually usable.

By the way; we’ve seen several community efforts of SharePoint 2007 treeview implementations that can be downloaded for free. If you can inspect the source code of such initiatives you should definitely take the time to do so. The ones we’ve seen use one of the approaches described above. The performance of such implementations will break in a harsh way if you want to add indicators for folders containing sub items (a very common request bound to be asked for by your end users eventually).

Train of thought #5: what about caching?

What options did this leave us? It was still very tempting to call SharePoint stored procedures directly. The other, most obvious, option is to implement a caching strategy. This specific solution has a few traits that doesn’t make it particularly easy to implement a caching strategy, especially if we‘d want to leverage the ASP.NET cache object to do so. For instance:

Since different users may have different permission levels we’d probably end up creating caches per user (or at best, caches for groups of users).
We needed to pass parameters to the cache and retrieve information based on those parameters (“give me all children of [url]”).
We needed to cache a lot of data. After doing some calculations for a specific small-to-medium size implementation we estimated that we needed to cache +/- 150 MB information (consisting of the names and URL’s of lists, folders, documents and adding a structure that allows parametric search).
Content might need to be cached for a long time. Building up the cache is a resource intensive process. Rebuilding it completely every 5 minutes, or even every hour puts a considerable strain on the server farm.
Content changes frequently. Every time a user removes or moves a SharePoint item the cache needs to be updated.

Our initial experiments that leveraged the ASP.NET cache were not particularly encouraging where in fact caching was not helping at all (speed-wise), which brings us to the next train of thought.

Train of thought #6: we need speed, we want velocity!

One of the things that occurred to us was that we needed to come up with a smarter caching strategy. If you search for “cache algorithms” you’ll find lots of different caching strategies out there that might help to improve performance. It also occurred to us that we needed an advanced caching mechanism that would be able to cater our demanding needs. That’s where a new MS product code-named “Velocity” enters the discussion (see http://blogs.msdn.com/velocity for more information).

Right now, it looks like Velocity will eventually be named the “Microsoft Distributed Cache”. This product provides a distributed caching solution for all types of .NET applications, although we see it as Microsoft’s implementation for Enterprise caching. If we understand the product’s team vision correctly, eventually Velocity will become a part of the .NET framework. The main traits of Velocity are that it provides a fast and scalable caching solution. As a side note, although Velocity is a product that can get quite complex and is highly configurable, we found it to be surprisingly easy to set up and get going.

In the rest of the article, we’ll use a simple caching algorithm and leverage Velocity to support our treeview scenario. However, before we start discussing such a solution there is some terminology that we need to get out of the way. After reading the next section, you should feel familiar with the following terms:

Cache host service.
Cache server.
Cache cluster.
Named cache.
Region.
Tags.
Cached objects.

Once you feel comfortable with these terms, we promise it’ll be easy to follow our code example.

Velocity runs on one or multiple servers as a Windows service which is named the cache host service. Every single server in a Velocity farm is called a cache server. All servers in a Velocity farm are called a cache cluster. The cache cluster can be used to store and distribute data and is managed by a couple of lead hosts. In this article we’ll keep things easier and use the most simple configuration by creating a single cache host service.

Before you’re able to store data in a cache cluster you’ll need to create (at least one) named cache first, a configurable unit of in-memory storage. Please note: there’s a default cache available called “default”, but we prefer to keep things organized by creating specific named caches for specific purposes.

After you’ve created a named cache, you can add regions to them. Regions are a very useful construct that allow you to search all cached objects by using descriptive strings called tags. You can associate one or more tags to a cached object in a region and the great thing about tags is that information in a tag is indexed. Regions have to be created explicitly at runtime using application code and we’ll definitely use this construct in our code example (which follows later). Please note: in this version regions are limited to a single cache host service and thus cannot be load balanced.

You can add .NET objects to named caches, or to regions in named caches, just as long as they inherit from System.Object. This means that you can store any object directly in cache. Once added, objects stay cached until the object time-out value (specified in app.config settings) is reached.

At the time of writing, the most recent version is the CTP 2 version (although the team blog mentions that CTP 3 is bound to be released very soon), which can be downloaded here: http://www.microsoft.com/downloads/details.aspx?FamilyId=B24C3708-EEFF-4055-A867-19B5851E7CD2&displaylang=en).

Please note: It’s better to install this version on your C: drive, otherwise the “Administration Tool - Microsoft Distributed Cache” PowerShell shortcut is pointed at the wrong location.

Although the options provided by Velocity are vast, there’s not much you need to do before you’re able to use it. First of all, you need to click on the “Administration Tool - Microsoft Distributed Cache” PowerShell shortcut which opens the Velocity command prompt.

Then, you need to start the cache cluster by typing the following command:

start-cachecluster

Once that’s finished, you need to create a new named cache. We’ll call ours “MossAppCache”. You can do this by typing the following command:

new-cache -CacheName MossAppCache -Secondaries 1

After doing this, you can check the results by issuing the following command:

get-cache

Now you’re already set to go and create an application that uses the Microsoft Distributed Cache. We’re creating a C# console application to test the performance of Velocity.

Make sure to reference the following Velocity assemblies (in CTP 2 stored at C:\Program Files\Microsoft Distributed Cache\V1.0\):

CacheAdminLibrary.dll
CacheBaseLibrary.dll
ClientLibrary.dll

Then, import the System.Data.Caching namespace. You can get a reference to the named cache created earlier (called “MossAppCache) using the following code:

ServerEndPoint[] servers = new ServerEndPoint[1];

//Specify Cache Host Details
// Parameter 1 = host name
// Parameter 2 = cache port number
// Parameter 3 = cache service name
servers[0] = new ServerEndPoint("[server name]", 22233, "DistributedCacheService");

//Select Routing or Simple Client
// True = Routing Client
// False = Simple Client (no routing table)
bool routingClient = true;

//Select local cache if desired
// True = Enable local cache
// False = Disable local cache
bool localCache = false;

//Pass configuration settings to cacheFactory constructor
//CacheFactory mycacheFactory = new CacheFactory(servers,
// routingClient, localCache);
CacheFactory mycacheFactory = new CacheFactory();

//Get reference to named cache called "MossAppCache"
Console.WriteLine("get cache");
Cache myDefaultCache = mycacheFactory.GetCache("MossAppCache");

Then create a region inside the named cache:

try
{
myDefaultCache.RemoveRegion("testregion");
}
catch
{
// No region found, but that's ok.
}

// Pass true if you want to allow the eviction of cached objects.
myDefaultCache.CreateRegion("testregion", false);

After that, it’s time for some good ole interaction with the SharePoint object model. We’ll be filling the cache with the contents of every site, every list and every folder we’re interested in. The following code is a simplified version that adds the contents of a specific list to the Velocity cache.

Console.WriteLine("add cache");

string strSiteUrl = "[absolute site coll url]";
string strWebUrl = "[relative web url]";
string strList = "[site coll relative list url]";

using (SPSite objSite = new SPSite(strSiteUrl))
{
  using (SPWeb objWeb = objSite.OpenWeb(strWebUrl))
  {
    SPList objList = objWeb.GetList(strList);
    IterateFolder(objList.RootFolder, myDefaultCache);

    // Add to cache.
    Tag tgIsRoot = new Tag("IsRoot");
    Tag tgHasChildren = new Tag("HasChildren");
    Tag[] arrTags = { tgIsRoot, tgHasChildren};

    myDefaultCache.Put("testregion", objList.RootFolder.Url,
    objList.RootFolder.Url, arrTags);
  }
}

The IterateFolder() method loops through each folder it can find and adds more entries to the Velocity cache. A simplified implementation of this method is shown below:

static void IterateFolder(SPFolder objFolder, Cache objCache)
{
  foreach (SPFolder objSubFolder in objFolder.SubFolders)
  {
    IterateFolder(objSubFolder, objCache);
    Console.WriteLine(objSubFolder.Name);

    Tag tgParentFolder = new Tag(objSubFolder.ParentFolder.Url);
    Tag[] arrTags = { tgParentFolder };
    objCache.Put("testregion", objSubFolder.Url, objSubFolder.Url,
    arrTags);
  }

  // Add current folder to cache.
  foreach (SPFile objFile in objFolder.Files)
  {
    // Add to cache.
    // In this particular implementation, we’re not interested
    // in adding files to the cache (we only need to do this once
    // we want to display files in the treeview as well.
    Console.WriteLine(objFile.Name);
  }
}

This time, we’ll be retrieving results from MS Distributed Cache instead of interacting with the SharePoint object model. We’ll get the children of a specific folder containing 2000 child items. This is shown in the following code:

Stopwatch objTimer = new Stopwatch();
objTimer.Start();

Tag objTag = new Tag("MyListName/FolderWithManyChildren/FolderWith2000Children");

List> objResults = myDefaultCache.GetByTag("testregion", objTag);
for (int i = 0; i < objResults.Count; i++)
{
  KeyValuePair objPair = objResults[i];
  Console.WriteLine("key: " + objPair.Key + " value: " +
  objPair.Value);
}

objTimer.Stop();

Console.WriteLine(String.Format("With caching::: Elapsed secs = {0}, in msec = {1}",
objTimer.ElapsedMilliseconds / 1000, objTimer.ElapsedMilliseconds));
Console.WriteLine("end of test");

You can add all pieces discussed above to a single console application and do some testing of your own. The simplified version of our test console application looks like this:

using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.Caching;
using Microsoft.SharePoint;

namespace VelocityTest
{
class Program
{
  static void IterateFolder(SPFolder objFolder, Cache objCache)
  {
   foreach (SPFolder objSubFolder in objFolder.SubFolders)
   {
    IterateFolder(objSubFolder, objCache);
    Console.WriteLine(objSubFolder.Name);

    Tag tgParentFolder = new Tag(objSubFolder.ParentFolder.Url);
    Tag[] arrTags = { tgParentFolder };
    objCache.Put("testregion", objSubFolder.Url,
    objSubFolder.Url, arrTags);
   }

   // Add current folder to cache.
   foreach (SPFile objFile in objFolder.Files)
   {
    // Add files to cache if you want.
    Console.WriteLine(objFile.Name);
   }
  }

  static void Main(string[] args)
  {
   // Use following statements to get Velocity up and running:
   // 1) start-cachecluster , do this if the cache cluster has
   // been stopped since the last time you ran code.
   // 2) new-cache -CacheName [naam] -Secondaries 1 , do this if
   // you need to create a new named cache.
   // 3) get-cache , retrieve info about the cache cluster.
   //
   // The default service port of Velocity is 22233.
   // The default cluster port of Velocity is 22234.
   // Our cluster name is MossApps.
   // Our named cache is called MossAppCache.
   // Our Region is called testregion.

   try
   {
    ServerEndPoint[] servers = new ServerEndPoint[1];

    //Specify Cache Host Details
    // Parameter 1 = host name
    // Parameter 2 = cache port number
    // Parameter 3 = cache service name
    servers[0] = new ServerEndPoint("asrv0243", 22233, "DistributedCacheService");
    //Select Routing or Simple Client
    // True = Routing Client
    // False = Simple Client (no routing table)
    bool routingClient = true;

    //Select local cache if desired
    // True = Enable local cache
    // False = Disable local cache
    bool localCache = false;

    //Pass configuration settings to cacheFactory constructor
    CacheFactory mycacheFactory = new CacheFactory();

    //Get reference to named cache
    Console.WriteLine("get cache");
    Cache myDefaultCache = mycacheFactory.GetCache("MossAppCache");

    Stopwatch objTimer = new Stopwatch();
    objTimer.Start();

    try
    {
     myDefaultCache.RemoveRegion("testregion");
    }
    catch
    {
     // No region found, but that's ok.
    }
    myDefaultCache.CreateRegion("testregion", false);

    Console.WriteLine("add cache");

    string strSiteUrl = "[URL of site collection]";
    string strWebUrl = "[relative URL of web]";
    string strList = "[site coll relative URL of list]";

    using (SPSite objSite = new SPSite(strSiteUrl))
    {
     using (SPWeb objWeb = objSite.OpenWeb(strWebUrl))
     {
      SPList objList = objWeb.GetList(strList);
      IterateFolder(objList.RootFolder, myDefaultCache);

      // Add to cache.
      Tag tgIsRoot = new Tag("IsRoot");
      Tag tgHasChildren = new Tag("HasChildren");
      Tag[] arrTags = { tgIsRoot, tgHasChildren };

      myDefaultCache.Put("testregion", objList.RootFolder.Url, objList.RootFolder.Url, arrTags);
     }
    }

    Stopwatch objTimer2 = new Stopwatch();
    objTimer2.Start();

    Tag objTag = new Tag("MyListName/FolderWithManyChildren/FolderWith2000Children");

    List> objResults = myDefaultCache.GetByTag("testregion", objTag);
    for (int i = 0; i < objResults.Count; i++)
    {
     KeyValuePair objPair = objResults[i];
     Console.WriteLine("key: " + objPair.Key + " value: " +
     objPair.Value);
    }

    objTimer2.Stop();

    Console.WriteLine(String.Format("With caching::: Elapsed secs = {0}, in msec = {1}",
    objTimer.ElapsedMilliseconds / 1000, objTimer.ElapsedMilliseconds));
    Console.WriteLine("end of test");

    Console.ReadLine();
   }
   catch (Exception err)
   {
    Console.WriteLine(err.Message);
   }
  }
}
}

Please note that in our lists, as was the case with our previous tests, we’re making sure none of the folders contains more than 2000 child items. Our test results tell us that, using Velocity, we can retrieve the children (including information if a folder contains child items) of any random folder within the 300-500 msec range.

Train of thought #7: Concluding the story

We had to travel long and far, but eventually found a way to fulfil our needs. Our last train of thought finally provided us with a viable option for implementing a treeview that’s responsive and provides all required functionality. This solves our Spider-Man/Human Torch dilemma where Velocity bears the burden of caching an average sized SharePoint content structure in order to provide the user experience we need. In an attempt to get the end deftly woven into our article there’s only one thing left to say: “And thus the journey concludes”.

Red alert: configuring Report Server in SharePoint Integrated Mode ain't funny all the time

04 May 2009 10:08:31

Red alert: configuring Report Server in SharePoint Integrated Mode ain't funny all the time

Written by: Margriet Bruggeman, Nikander Bruggeman.

Red alert: configuring Report Server in SharePoint Integrated Mode ain't funny all the time

We're writing this because we're planning to never ever ever, when configuring Report Server in SharePoint integrated mode, get stuck on the following error message:

“The report server cannot access settings in the SharePoint configuration database. Most likely, the Windows SharePoint Services object model is not installed or the Report Server Web service and Windows service accounts do not have access to SharePoint database, To configure service account access, use SharePoint Central Administration”

This message may appear when configuring Report Server using the Reporting Services Configuration Manager. What do you do when you have successfully created a Report Services database that runs in SharePoint Integrated mode and you're still getting a red x sign next to the SharePoint Integration tab?

First, you need to check a couple of things:

Make sure that you’ve installed SQL Server SP 2 (http://technet.microsoft.com/en-us/sqlserver/bb426877.aspx) and the SQL Server 2005 Reporting Services Add-in for Microsoft SharePoint Technologies (http://www.microsoft.com/downloads/details.aspx?familyid=1E53F882-0C16-4847-B331-132274AE8C84&displaylang=en) . You can check the version of SQL Server via the following query: SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition') More about identifying SQL Server version and edition can be found at the following location: http://support.microsoft.com/default.aspx?scid=kb;en-us;q321185.
Verify that you are site collection administrator for SharePoint Central Administration. You can do this by browsing to the SCA > Site Actions > Site Settings. If you are a site collection administration you will be able to see a section called “Site Collection Administration”.
While you are at the site settings page of the SCA make sure that the “Reporting Server Integration Feature” is activated. In the “Site Collection Administration” section click “site collection features”. This page shows a list with features that are available for the site collection. Locate the “Reporting Server Integration Feature” feature and click Activate if this feature is not already activated.

Then, when you are 100% sure that the SharePoint object model is installed, the Report Server Web service and the Windows service account have access to the SharePoint database and you've installed SQL Server 2005 SP 2 and the SQL Server 2005 Reporting Services Add-in for SharePoint, but you still can’t get rid of the red x which really starts to annoy you, then and only then, it's time to apply the magic move:

Go to the Reporting Services Configuration Manager.
Select the Server Status tab.
Stop and start the Report Server Windows service.

If it works, power to the magic move!

Clap and yell: Warm up SharePoint

28 February 2009 11:15:56

Written by: Margriet Bruggeman, Nikander Bruggeman.

Clap and yell: Warm up SharePoint

It is rumoured that, in the early days of his career, a famous Texan movie star with big muscles used to perform in a sleazy circus to make some money. He used to perform a number of weird acts such as drinking large amounts of beer in front of an audience. The one act we thought was most curious of all was called “Clap and yell”. During this performance he used to climb to a considerable height and then jumped on a trampoline whilst doing what the title of the act promised: clapping and yelling during his fall. This blog post is much like that, in the respect that it delivers exactly what the title promises.

Numerous blog posts have been written about warming up SharePoint, but the one blog post that always comes up is called Warm up your SharePoint Servers… by Joel Oleson. He explains why you’d want to warm up your SharePoint servers and has created a solution that doesn’t seem to be downloadable from the web site anymore, although you can find it on an alternative location.

There are a couple of other interesting links that are worth mentioning:

The APM blog post discusses a developer tool created by Spencer Harbar, the Application Pool Manager, that warms up a SharePoint server. This one is only meant as a developer tool, and can’t be used to warm up production environments.
Andrew Connell has created a solution that uses SharePoint timer jobs to warm up SharePoint quietly in the background.
We seem to recall that CodePlex hosts several solutions, such as SPWakeUp.
Kirk Hofer has provided a PowerShell script that is very easy to use.
Other people, such as long-time SharePoint MVP Mike Walsh, disagree respectfully and doubt the need to do a warm up altogether.

All posts provide interesting reading and a good way to start when finding a warm up solution that suits your needs. However, none of the solutions mentioned quite fitted our needs, most were overkill of some sort. We only wanted to warm up a small set of specific pages and wanted to keep things real simple to install, troubleshoot and maintain.

That’s when we stumbled upon a blog post written by Brad Smith called Poor man’s SharePoint warm-up script. Immediately, we were enthusiastic about the utter simplicity of the idea of creating an HTML page that uses IFrames to warm up pages.

This allows you to upload the HTML page to a document library in a production environment, check that all parts work as suspected by manually opening the page and maintain the list by downloading it, modifying it and uploading it again. We feel this approach has a couple of major advantages:

It’s very easy to set up.
It’s very easy to maintain and monitor.
It allows you to see immediately if a warm-up URL delivers the anticipated gains in speed as perceived by the end user.
It doesn’t require a server-based configuration file which can be a huge advantage when working in a large corporate environment where you need to pass an elaborate change request process before you’re allowed to add a new URL that needs warming up to such a config file.

Because we liked the idea so much, we’ve started experimenting with it and modified it a bit so that the page containing the IFrames automatically refreshes itself automatically after a predefined amount of time has elapsed, like so:

<html>
  <head>
    <title>Warmup</title>
    <!—
    Refresh time in seconds
    -->
    <meta http-equiv="refresh" content="60" />
  </head>
  <body>
    <h1>Warmup page</h1>

    <iframe src="http://URL1" />
    <iframe src="http://URL2" />
    <iframe src="http://URL3" />
    <iframe src="http://URL4" />

  </body>
</html>

This results in a warm up page that looks like this:

Unfortunately, we didn’t understand what Brad Smith meant with his mention of warmup.cmd. Instead, we wrote a small .NET console application that takes care of calling the page containing the IFrames. The console application uses the WebClient class to download the contents of the HTML page and then uses regular expressions to filter out the URLs of the IFrames and finally loads those pages. The next code listing shows how to create this console application:

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;

namespace WarmUp
{
  class Program
  {
    static void Main(string[] args)
    {
      try
       {
         Console.WriteLine("Start download warmup page.");

        string strUrl = args[0];
        string strResult = DownloadUrl(strUrl);
        DownloadIFrames(strResult);
      }
      catch (Exception err)
      {
        Console.WriteLine("Warmup failed because " + err.Message);
        throw;
      }
    }

    static void DownloadIFrames(string strHtml)
    {
      string strRegExpr = "iframe src=\"(?'URL'.*)\"";

      Regex objReg = new Regex(strRegExpr, RegexOptions.IgnoreCase);
      MatchCollection objMatches = objReg.Matches(strHtml);

      foreach (Match objMatch in objMatches)
      {
        try
        {
          string strUrl = objMatch.Groups["URL"].Value;
          Console.WriteLine(DownloadUrl(strUrl));
        }
        catch (Exception err)
        {
          Console.WriteLine(String.Format("Failed to download {0} because of {1}", objMatch.Value, err.Message));
        }
      }
    }

    static string DownloadUrl(string strUrl)
    {
      WebClient objClient = new WebClient();
      objClient.UseDefaultCredentials = true;
      return objClient.DownloadString(strUrl);
    }
  }
}

You can schedule this console application and make sure to do so on every WFE.

So, there you have it, yet another way to warm up your SharePoint server!

Exploring the Data View Web Part

19 February 2009 07:42:50

Written by: Margriet Bruggeman, Nikander Bruggeman.

Exploring the Data View Web Part

Companies typically use a multitude of repository types to store and manage their data. A company may use SQL Server to store relational data; the Windows file system for storing semi-structured data and eXtensible Markup Language (XML) files to hold hierarchical data. The need to aggregate and manage data in a central location is a very common requirement within portal environments. To cater to this need, you could build custom Web Parts using Visual Studio.NET to incorporate various data sources. Instead, for a range of scenarios, you will want to try to use the Data View Web Part first. The Data View Web Part is an advanced tool that allows you to create solutions for viewing and managing data in a fraction of the time it would take you to build a similar solution in Visual Studio.NET. The Data View Web Part lets you view and manage data coming from different data sources, like Web services, SharePoint lists and server-side scripts.

Data View Web Parts are able to retrieve data from various data sources in the form of XML even if the data itself in its original form is not XML, and make it very easy to adjust the appearance of that data by applying eXtensible Stylesheet Language Transformations (XSLT) to it. XSLT is used for transforming the structure of an XML document. The XML data within a Data View Web Part can be formatted using the Microsoft Office SharePoint Designer’s Design view. The next procedure explains how to open a SharePoint page in design view using a Internet Explorer.

Open Internet Explorer and navigate to a SharePoint site.
Click the Page button at the upper right corner. If SharePoint Designer is installed on your machine, this opens a menu that contains the option Edit with Microsoft Office SharePoint Designer. Click Edit with Microsoft Office SharePoint Designer, which by default, opens the SharePoint site in SharePoint Designer’s Design view.
If the page is opened in another view, you can switch back to Design view by clicking the Design tab at the bottom.

The Data View Web Part offers many possibilities, such as consuming various data sources, sharing data sources, defining the look and feel of data overviews, and adding editing capabilities to data overviews. The Data View Web Part is all about aggregating and managing data from various data sources. The first thing you need to learn about the Data View Web Part is how to import and display data. To start this discussion, we will take a closer look at data sources, data source libraries, and SharePoint Designer.

Importing and displaying Data

From the perspective of the Data View Web Part, a data source is a repository of information or an end point that provides access to an information repository, for example, a database or a Web service. The Data Source Library is the main entry point for accessing and managing data sources within SharePoint sites.

SharePoint Designer, based on FrontPage technology, is a powerful tool that makes both Windows SharePoint Services (WSS) 3.0 and Microsoft Office SharePoint Server (MOSS) 2007 implementations a lot easier. SharePoint Designer lets you define new data sources, add Data View Web Parts to SharePoint Web pages that access, and display these data sources in a visual way. Manipulating the Data View Web Part visually might seem to have to do less with software development when compared to typing code yourself, but the fact of the matter is, nowadays creating software becomes more and more abstract and visual.

Using the Data Source Library

In our experience, most developers oversee the Data View Web Part and create a custom solution to access a data source. This often happens because a developer is more familiar with Visual Studio.NET then SharePoint Designer. It is advisable to add the Data View Web Part (and SharePoint Designer) to your bag of tricks because it can be a real time saver.

The Data View Web Part allows people who are not developers to manipulate data sources. Although we believe the Data View Web Part is easy enough to use so non-developers can use it to manipulate information, this does not seem to happen a lot at the companies we visit. Creating gateways to data sources is often considered the responsibility of developers, which excludes other types of users from using the Data View Web Part.

To use a Data View Web Part to display data, a reference to that data source must be referenced in the Data Source Library task pane. The Data Source Library offers access to data that is stored within SharePoint Web sites, as well as external data sources. As long as you have access rights to a data source, you can manage it via the Data Source Library task pane. In the Data Source Library task pane under the Current Site section, you will see lists and libraries that the site contains, plus any other data sources you have attached. The data sources are grouped under section headers where the data sources are accessed using the same method, with the number of currently added data sources in brackets.

SharePoint Designer automatically creates data sources for the default lists and libraries that exist within a given SharePoint site. Out of the box, entries for the following lists are created in the Data Source Library and added in the SharePoint Lists section: Announcements, Calendar, Links, Tasks, and Team Discussion. There is also an entry for the Shared Documents document library, which is listed under the SharePoint Libraries section.

Creating a new SharePoint list or library using Data Source Library

SharePoint lists and libraries are used within SharePoint implementations to not only store information directly targeted at the end user, for instance, documents, tasks, and events. Lists and libraries can also be used to store configuration information or information required to provide a feature. You can also create lists to store your own configuration information and reuse this information as metadata for documents that you add to a document library. As we have already seen, SharePoint Designer automatically adds references to lists and libraries in your current site, so you do not have to add manually references to these lists and libraries yourself. Once you have a reference to a list or library as a data source, the Data View Web Part lets you display the SharePoint list or library in a different way. You can also link the list or library to another data source, thereby allowing you to merge or join data from several data sources; more about this later in this article.

However, what the Data Source Library task pane, does allow you to do, is to create new lists and libraries. Follow the next procedure to create a SharePoint list as a data source.

Open the Data Source Library task pane, click the Data View menu and then click Manage Data Sources.
Click the + sign to the right of SharePoint Lists.
Click Create new SharePoint list.
On the SharePoint Content tab in the New dialog box, click Lists. SharePoint Designer contacts the current SharePoint Web site and asks for a list of lists templates that you can use as a basis for your new list. SharePoint Designer then displays these list templates in the New dialog box.
Click one of the list templates, enter a name for your new list in the Specify the name for the new list text box and click OK. SharePoint Designer uses the name you enter for the URL and the Title of the list.

Creating a SharePoint library as a data source is almost the same as creating a SharePoint list. Follow the next procedure to create a SharePoint library from the Data Source Library task pane.

An alternative way of opening the Data Source Library task pane is to click the Task Pane menu and click Data Source Library.
Expand the SharePoint Libraries section by clicking the + sign.
Click Create new SharePoint libraries.
On the SharePoint Content tab in the New dialog box, select Document Libraries. Again, SharePoint Designer asks the current SharePoint Web sites for a list of library templates, which will then be displayed in the New dialog box. Although the link caption says Document Libraries, all kinds of libraries templates site are listed. For instance, the Document Library section can also contain the form or wiki page library templates.
Select the library you want to create. Enter a name for the new list data source in the Specify the name for the new document library text box, again remember that this is the URL and the Title for the Library, and then click OK.

You have now created a new list and library using the Data Source Library task pane, and SharePoint Designer automatically added references to those lists and libraries in the Data Source Library task pane. Next, we will add references to other data sources so that they appear in the Data Source Library task pane.

Adding a Database as a data source

It is good practice to store relational data in databases. Most companies that use SharePoint technologies to build portals sometimes need to reuse this data within the portal. For example, we recently had to expose customer information stored in a relational database on a portal Web page. We had to resist our first instinct to grab Visual Studio.NET and create a new SharePoint Web Part, and ended up saving a couple of hours of our time by using the Data View Web Part instead. If you find yourself in a situation where you need to reuse data stored in a relational database, you should start by adding the database as a data source by specifying a database connection for it. After that, you are free to use the data from the database within the Data View Web Part. The next procedure shows you how to add a database as a data source.

Open the Data Source Library task pane using one of the methods we described in the following section.
Expand the Database Connections section, and click Connect to a database, which opens the Data Source Properties dialog box.
Select the General tab and enter a descriptive name for the database connection, so you will know when you see the name what database it refers to.
Select the Source tab and click Configure Database Connection button. The Configure Database Connection dialog box is displayed.
Type the name of the database server in the Server Name text box.
Choose the type of authentication you want to use, by selecting one of the following options:
- Save this username and password in the data connection. Here you can specify a username and password that will be saved in the data connection. Selecting this option will save the specified username and password as clear text in the data connection, which poses a security risk. A dialog box is displayed warning you of this. If the database server is using Microsoft SQL server, then to use this option, the SQL Server is configured for mixed authentication, that is both Windows authentication and SQL authentication. The user name you specify is an SQL username and not an Active Directory (AD) user name. If you are using this option, you should persuade your IT department to implement some secure method of transferring data between your SharePoint servers and the SQL database, such as IP Sec or VPN.
- Use Single Sign-On authentication. This option is only available when the SharePoint site is using a Shared Services Provider (SSP), which is part of MOSS 2007, and your Shared Services Administrator has enabled and configured single sign-on. When you click the Settings button the Single Sign-On Settings dialog box opens, where you can specify the Application Name, the Application field to use as the username and the Application field to use as the password in the appropriate text boxes.
- Use custom connection string. Here you can specify an OLEDB connection string to connect to the database. Click the Edit button to open the Edit Connection String dialog box where you can specify the provider name and the connection string. You will probably need the help of your database administrator for this option.
- Windows Authentication. You will see this option if your organization has installed only WSS 3.0 on a single server, and SQL Server is installed on the same server as WSS 3.0.
Click Next when you have configured the authentication method. The Configure Database Connection dialog box is displayed.
Select a database from the Database dropdown list.
If you want to use a table or view, select the Select a table or view radio button and select the table or view from the list box. It is also possible to create your own query by checking the or select custom Select, Update, Insert, and Delete commands using SQL or stored procedures radio button.
Click Finish. This brings you back the Data Source Properties dialog box.
Click OK to finish the creation of a database connection data source.

Now that you have added a database as a data source, you are ready to use it within the Data View Web Part. We like this option and use it a lot, although maybe not as often as we use a SharePoint list or library as a data source within the Data Source Library task pane.

Adding an XML file as a data source

XML files are commonly used in organizations because they are very suitable for storing hierarchical data. We remember a company that hired us to build a Web-based form generator. The company used a relational database model to describe medium-sized (around 100 fields) form definitions. If a visitor came to their Web site and submitted one of those forms, the data was added, again, to a relational database model. It would often take literally hundreds if not thousands of separate SQL queries to build a form based on a definition and add the user data to the database.

We were able to improve the speed of constructing forms and adding data to a repository enormously by switching to an XML repository. Not only did the performance of the Web-based form generator increase several times, reducing the number of required SQL queries to a maximum of two, the upper limit of supported fields in a form increased from around a hundred fields to tens of thousands of fields. Once you reach Web-based forms of that size, limits do not count anymore, since you will not be able to find a user who is prepared to take a week of his or her life to fill-in and submit such a form. By the way, we were not the only ones who has come up with the idea to store form definitions and form data in XML; Microsoft’s InfoPath is an excellent example of a tool that does just that.

When SharePoint Designer first opens the Data Source Library task pane it will search your current site for any XML files and will list them under the XML Files section. To add other XML data sources to your SharePoint site follow the next procedure.

Open the Data Source Library task pane as described previously, if it is not already open.
Expand the XML Files section and click Add an XML file. This opens the Data Source Properties dialog box.
Click Browse, to open the File Open dialog box, and select the XML file you want use as a data source. Next, a warning popup window appears asking if you would like to import the specified XML file in order for it to be used as a data source. Click OK.
In the Import dialog box click OK. Now you have uploaded an XML file to the root of your SharePoint site and a data source is added to the XML Files section.

You can move the XML file to one of your document libraries, and SharePoint Designer will amend the data source accordingly. By moving the XML file into a document libraries, means that you can edit the XML file using the browser, whereas by leaving the XML file in the root of the SharePoint site, then only users who have access to SharePoint Designer can modify it.

XML files are great for solving specific types of problems, as you have seen earlier in this section. It is not hard to add an XML file as a data source, in fact, it is not much different than the types of data sources you have seen earlier in this article.

Server-side script

Using the Data Source Library you can connect to server-side scripts that are written in different programming languages like ASP.NET, and PHP. Server-side scripts have been used for years and play an essential role when it comes to generating dynamic Web sites. There is only one important condition when it comes to consuming server-side scripts: the response of a server-side script needs to be valid XML. If you want, you can use the Internet Explorer Developer Toolbar to determine if a page is valid eXtensible Hypertext Markup Language (XHTML). Any page that is XHTML compliant is valid XML and thus can be used as a data source. You can download the Developer Toolbar add-in from Microsoft’s download Web site: http://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en .

Really Simple Syndication (RSS) feeds form a very important and popular source of delivering information that fall under the server-side script umbrella. If you have a Web site that does not provide RSS feeds, it will definitely cost you many visitors. On our Web sites, RSS feeds are responsible for about 70% of the traffic. If the extension of an RSS feed ends with .XML, it can be added as a data source in SharePoint Designer, via either the server-side script category or XML file category. Any RSS feed that ends with another extension, for instance, .aspx, you can only add as a data source using the server-side script category. The next procedure shows you how to add a server-side script or RSS feed as a data source.

Open the Data Source Library task pane as described previously, if it is not already open.
Expand the Server-side Scripts section and click Connect to a script or RSS feed. This opens the Data Source Properties dialog box.

Click the General tab and specify a name for the data source.
Click the Source tab and select which HTTP method you want to use. You can choose between HTTP Get and HTTP Post. The drawback of the HTTP Get method is that all parameters are shown in the URL itself. In general, if you are working with a set of parameters of a medium size or larger, choose to use the HTTP Post method.
Select which data command you want to configure. You can choose between Select, Insert, Update or Delete.
In the Enter the URL to a server-side script text box, enter the URL of the server-side script or RSS feed you want to connect to.
In the case where you are using a server-side script, which requires parameters, click the Add button to add a parameter. This opens the Parameter dialog box where you can enter a name and default value. Click OK in the Parameter dialog box.
Click OK in the Data Source Properties dialog box to add the server-side script or RSS feed as a data source.

At first sight, you might not look at a server-side script as a potential data source. Nowadays server-side scripts return responses that are better structured than they used to be. Server-side script responses that are well structured are suitable to be consumed as a data source by the Data View Web Part.

XML Web service

In the current development landscape, using Web services is, without a doubt, the most popular technique for implementing Service Oriented Architectures (SOA). Web services make it easy to issue Remote Procedure Calls (RPCs) over the Internet or a network using standards such as HTTP and XML. The messages exchanged between a client and a Web service, are encoded in the Simple Object Access Protocol (SOAP), a special XML dialect. The interface of a Web service is described in a language called the Web Service Description Language (WSDL), which, states the names of each Web service method and the parameters it accepts and returns.

Because Web services have to adhere to the SOAP and WSDL standards, Web services are very interoperable. SharePoint Designer can consume any Web service built on any platform using any programming language, as long as the Web service complies with the SOAP and WSDL standards.

Because of the enormous popularity, it is good news that you can add a Web service as a data source. If you want to do this, you have to know the URL for the WSDL description of the Web service. This URL ends with either ?WSDL or .wsdl. WSS 3.0 provides an extensive Web services layer that allows you to interact with different aspects of WSS and is the primary means of remote communication with SharePoint. For instance, you can use the SharePoint Web service layer to interact with servers, lists, and sites. SharePoint Server 2007 provided additional Web Services so that you can remotely access information stored in components, such as, user profile and business data catalog. For a full list of the SharePoint Web Services, refer to the WSS 3.0 and SharePoint Server 2007 Software Development Kits (SDKs). In this section, we will use one of those Web services for demonstration purposes.

There are other ways to communicate with SharePoint remotely as well. You can use Web-based Distributed Authoring and Versioning (WebDAV) to manipulate documents on a SharePoint server. Then, there is FrontPage RPC, a protocol that is powerful when it comes to working with documents on a SharePoint server. Finally, there is SharePoint RPC, a protocol that contains a plethora of functions that fall outside the scope of this article. By the way, do not be overjoyed if you start a new project that uses either one of the WebDAV, SharePoint RPC, or FrontPage RPC protocols. Programming in .NET using these protocols is not that well documented and can be quite laborious.

In this example, we are going to demonstrate how to add a web service as a data source. We are going to use the SharePoint Lists Web service, which provides methods for interacting with SharePoint lists and list data. Let’s look at how to query the Lists Web service for all the titles of the lists in a SharePoint site.

Open the Data Source Library task pane as described previously, if it is not already open.
Expand the XML Web Services section and click Connect to a web service. This opens the Data Source Properties dialog box.
Click the General tab and enter a meaningful and descriptive name for the data source.
Click the Source tab and enter the URL for the Web service in the Service description location text box. Append to the URL name _vti_bin/Lists.asmx?WSDL. Your URL should look something like: http://[Servername]/_vti_bin/Lists.asmx?WSDL.
Click the Connect Now button to establish a connection with the web service.
After the connection is created, the Connect Now button will change into a Disconnect button.
Select the Select command in the Select which data command to configure list. The other commands are Insert, Update and Delete.
In the Port list select the application protocol you want to use to access the Web service. For .NET Web Services there are two protocols available: one with a SOAP 1.1 binding called [WebServiceName]Soap, and another with a SOAP 1.2 binding called [WebServiceName]Soap12. Those options are only shown if the Web service indicates that it is able to support the multiple versions of the SOAP protocol. If the option is available to choose between multiple versions, we advice you to choose the latest version.
In the Operations drop down list, you will find all the methods of the specified Web service. In this example, we select the GetListCollection method, which retrieves the names and Globally Unique Identifiers (GUIDs) for all the lists in the site. The Parameters list box can be used to display the names of any parameters that the Web service requires or accepts. After you select the Web service method, SharePoint Designer will fill the Parameters list box with the required parameters and their data values. If there are input parameters in the list you will have to supply these to the Web service. This can be done by hard coding the value of the parameter or by setting a parameter at run time.
The Lists Web service does not require any parameters, so you can just click OK. The next figure shows a Data View Web Part that uses the Lists web service as a data source. Later in this article, we will show you how to create Data View Web Parts.

A considerable number of software systems today are built with SOA in mind. Many of those systems will use Web services as the primary way to connect to other systems, and let other systems connect to it. Because of this, the ability to use a Web service as a data source within a Data View Web Part is an important one. More information about SOA can be found in the green book "Application Architecture for .NET: Desiging Applications and Services". This freely downloadable book can be found at the following location: http://www.microsoft.com/downloads/details.aspx?FamilyId=A08E4A09-7AE3-4942-B466-CC778A3BAB34&displaylang=en .

Business Data Catalog

The Business Data Catalog (BDC) is one of the new features of MOSS 2007. The BDC enables you to display information coming from various types of backend applications. The BDC can act as another type of data source for the Data View Web Part. Configuring a BDC requires some work, which falls outside the scope of this article.

Creating a Linked source

Organizations often store data in different types of repositories, for example, XML Files and relational databases. With the Linked Source section of the Data Source Library task pane, it is possible to relate different types of data sources to each other. In this way, you can combine several types of data sources into a single data source and then use the Data View Web Part to display that data. We discuss creating linked sources later in this article.

Connecting to another library

You are not limited to using the Data Source Library of the current SharePoint site; you can also use the data sources as defined in the Data Source Libraries from other SharePoint sites. This makes it easier to share data sources between sites without recreating data sources and we expect you will be using this feature all the time. In fact, we like creating SharePoint sites whose single purpose in life it is to act as a central repository for hosting data sources. Of course, this only makes sense if you are planning to use a data source in multiple other sites. Creating a central SharePoint site that acts as a data source host makes it easier to find and manage data sources.

When you connect to another Data Source Library, SharePoint makes all its data sources available in the current SharePoint site. This does not mean that you are moving or copying of data sources to your site, you are simply referencing an external Data Source Library. Any deletions or modifications in the external Data Source Library will also affect the current SharePoint site. The following procedure shows you how to connect to a Data Source Library in another SharePoint site.

Open the Data Source Library task pane.
Click the Connect to another library link at the bottom of the Data Source Library task pane. This opens the Manage Library dialog box.
Click the Add button in the Manage Library dialog box; this opens the Collection Properties dialog box.
In the Display Name text box enter some text that will help you to recognize the new data source within your Data Source Library.
Click Browse to browse for the SharePoint site that contains the Data Source Library you want to use. This opens the Choose a Web Site dialog box.
Select a web site or type the URL of the SharePoint site and click Open.
In the Collection Properties dialog box, click OK.
In the Manage Library dialog box, click OK. The figure below shows the Data Source Library task pane with the data sources of the Current Site and the data sources of another SharePoint site which we used a display name of Another SharePoint Site that contains the external Data Source Library.

Now you have now created a number of datasources, and you know how to define data sources in a central SharePoint Web site location, from where you can share these data sources with other SharePoint sites. However, before going on to use them we need to take some time out to find where all these data sources are physically stored, which is the subject of the next section.

Data Source document library

When you create a data source, SharePoint Designer creates a XML file that describes that data source and then stores the XML file in a document library, named fpdatasources. This is a special document library that you can only see when you use SharePoint Designer and the Folder List task pane. To view the fpdatasources document library complete the following steps:

Open the Folder List task pane: Task Panes > Folder List.
Expand _catalogs and fpdatasources. You should now see a list of XML files, which you can open and review the XML if you wish. You could change the XML in these files, but the next time you use the Data Source Library to amend your data sources, you will lose any amendment you enter manually, so we recommend you don’t do this.

All data sources you create are therefore stored in the SharePoint SQL content database and are therefore available not just to you, but to anyone who has the rights to use SharePoint Designer on your SharePoint Web site. Now we are ready to describe how to expose the data that these data sources contain, that is, in the next section we’re going to start to use the Data View Web Part.

Creating a Data View

Until now, you have learned how to add data source that points to various repositories so that the data in those repositories can be viewed or managed using the Data View Web Part. It is about time that you learn how to use the Data View Web Part, don’t you agree? It is only then that you will see the power of making data available as a data source. In this section, you will learn how to create a Data View that displays data coming from a data source.

The first thing you need to do when you start to create a Data View is to make sure you have added the data source to the Data Source Library. Adding data sources to a Data Source Library, we discussed in previous section. After ensuring the data source you need is present, you are ready to create a Data View. There are two ways to create a Data View we’ll discuss both of these methods in this section. The next procedure shows the first way to insert a Data view. This method uses the Folder List task pane to add a Data View.

Open SharePoint Designer and open the SharePoint page where you want to place the Data View Web Part.
In the Folder List task pane, on the left side of SharePoint Designer, locate your data source. This can be a list, a library, an XML file or one of the XML files in the fpdatasources document library. Select the data source and drag it onto the SharePoint page.
Two things will happen: a Data View Web Part containing the selected data source will appear onto the SharePoint page and the Data Source Details task pane opens. The Data Source Details task pane is responsible for displaying all the elements of the selected data source. You can also open the Data Source Details task pane by going to the Data Source Library and clicking the data source and selecting Show Data from the dropdown list.

The second method for creating a Data View uses the Data Source Details task pane, which we’ll explain in the next procedure.

Open SharePoint Designer and open the SharePoint page where you want to place the Data View Web Part.
On the Data View menu select Insert Data View. This will place a Data View Web Part onto your SharePoint page. Next, we are going to insert the data.
In the Data Source Library task pane, select the data source you want to add, click the data source and select Show Data. This opens the Data Source Details task pane.
In the Data Source Details task pane, select the fields you want to add. Like other Microsoft products, you hold down the SHIFT or CTRL keys to select multiple fields at the same time.
Click the Insert selected fields as dropdown list and select Multiple Item View.

If you just want to create a basic Data View Web Part then you are ready. If you want to customize your Data View Web Part there are plenty of options you can choose. In this section, we will discuss these options some more.

If you click the chevron icon to the top right of the Data View Web Part, you open the Common Data View Tasks action panel, where you will notice the following options.

Filter
Sort and Group
Paging
Edit Columns
Change Layout
Data View Preview
Show with sample data
Conditional Formatting
Web Part Connections
Parameters
Refresh Data View
Data View Properties

Probably the biggest advantage SharePoint has over its IBM and Oracle portal product competitors is its compelling user interface. The Data View Web Part offers advanced capabilities to create a user interface that looks good, and we think your customer expects you to use it.

Filtering Data

In most scenarios, it will not do to show all the data coming from a data source. You will typically want to filter this data. In SharePoint Designer, you can add a filter to the Data View Web Part. A filter allows you to specify a subset of all the data that your data source contains and in this way you also limit the data that are shown in the Data View Web Part. Like the List View Web Part, it is possible to add a filter toolbar to the Data View Web Part so that users themselves can filter the data and determine which subset of the data available via the Data View Web Part they can display.

Sorting and Grouping Data

SharePoint Designer makes it easy to sort and group data in a Data View Web Part. Sorting data in a Data View Web Part modifies the order in which you show the data. For instance, you can sort the data alphabetically to make it look more organized. You can create complex sort expressions using the XPath Expression Builder. You can find more information about the XPath Expression Builder later in this article.

When you group data in a Data View, data within the Data View is grouped by the criteria that you specify. Sorting and grouping are related, and you cannot group data unless you have specified a sort order. Each group in a Data View can be expanded or collapsed by using the group header.

The Data View Web Part displays the data sort or group order that you have specified. You can also add a toolbar to the Data View Web Part so that users can sort or group data themselves.

Paging

In a Data View Web Part, you can limit the number of records shown per page this is called paging. SharePoint Designer will add paging navigation to the Data View Web Part for moving backward and forward through all the data. This is one of the most asked features when it comes to displaying data, so we can promise you, you will use the Paging option a lot.

Editing Columns

If a customer is happy to use the columns that are shown by default in a Data View Web Part, he or she is probably very easy to deal with. We doubt you are that lucky, so you will be glad to learn that you can influence which fields are displayed by the Data View Web Part. You can add, move or remove the columns or rows that are shown using the option Edit Columns. This option is only present when you are working with a Data View Web Part that displays the data in an HTML table.

Changing Layout

The data of a Data View Web Part is by default displayed in a basic table layout. This layout can be changed quickly and easily by making use of the built-in Data View layouts, of which there are 13 available, each with a preview of the layout and a description of the style. Changing the style of a Data View Web Part will remove any custom formatting that you have done previously, so we recommend you do this first before applying any custom formatting.

Data View Preview

The Data View Preview option allows you to preview the data and offers you the possibility to show a limited the amount of data in the preview. The following options are available:

Default. Matches the view that is shown at runtime.
Hide all filters. Shows all available data.
Limit to 1 item. Shows a single item.
Limit to 5 items. Shows up to 5 items.
Limit to 10 items. Shows up to 10 items.
‘No Matching Items’ Template. Shows what happens if no matching items are found and allows you to specify text for this situation.

Show with sample data

Under the Data View Preview option there is a checkbox Show with sample data, which if selected, the Data View Web Part will fill with sample data instead of real data. This option proves to be very useful when a data source contains much data or when it takes a lot of time to retrieve data from a data source. The Show with sample data option is also convenient when the data source itself does not contain any data yet.

Conditional Formatting

With the Conditional Formatting option, it is easy to create a Data View Web Part that applies a style to a selected HTML tag or data value when the data meets specified criteria. It is possible to set conditions that change the visibility of an HTML tag or data value.

Web Part Connections

With the Web Part Connections option, you can create a connection between two Data View Web Parts and in this way, when you perform an action in one Data View Web Part, it will change the contents of another Data View Web Part. The only way to create a connection between two web parts on different pages in the same SharePoint site (cross-page connection) is using SharePoint Designer.

Parameters

You can specify parameters coming from different sources such as cookies, form values and query string variables. Any parameter you specify can be used when creating Web Part connections. The discussion of this falls outside the scope of this article.

Refresh Data View

With the Refresh Data View option, you can refresh the content shown in the preview. After a refresh, the latest data from a data source is shown.

Data View Properties

When you choose the Data View Properties option, the Data View Properties dialog box opens. Many of the other options described above open this dialog box, from which you to specify the look and feel, Paging and other options.

Creating a Read/Write Data View

You can add Data Views to a SharePoint page as either a view or a form. Forms allow you to edit the data at run time and the changes are written back to the data source. Views are great for providing overviews of data, but you can also add links to a view so that users can edit the data when they display the data in a Web page, also known as "at run time". The first thing you will learn in this section is how to create a view that displays single or multiple records. After that, you will see how to collect data from users using item forms. If you are unclear about all the differences between data views and data forms, we have provided a table as an overview of the differences between Data Views that are added to SharePoint pages as views and Data Forms where you can manage the data.

Actions	View	Form
View style	Data is displayed in view mode and by clicking the Edit link it will be displayed in Edit mode.	Data will always be displayed in Edit mode.
Insert	X	X
Edit	X	X
Delete	X
Data Sources
Lists and libraries	X	X
Database connections	X	X
XML files	X	Only local XML files
Server-side scripts	X
XML Web Service	X
Business Data Catalog	X
Linked source	X

Views are similar to the ASP.NET GridViews component. They are great for providing dense overviews of multiple rows of information. Forms are similar to the ADO.NET FormViews component. They show one or more rows of data, records, at a time, are always shown in edit mode, and are very customizable thus providing optimal flexibility.

Creating Item Views

SharePoint Designer provides two options for inserting a data view as a view.

Single Item View. This view displays a single record as a view. Optionally, you can provide navigation links that allow you to view the next or previous record.
Multiple Item View. This view displays multiple records as a view.

It is possible to create read-only views or you can add editing links that allow you to edit, insert or delete records from the data view. Multiple Item Views are less suitable for implementing advanced editing scenarios. Displaying and editing multiple items at a time tends to confuse users. In general, this is solved using master-detail views, which you can achieve by connecting two Data View Web Parts, as discussed briefly previously in this article.

Adding a data view as a form is described in the next section. You can also add editing links to a form by replacing the existing form buttons. This effectively transforms a form to a view.

In the next example, we are going to use the SharePoint Announcements list as a data source for our Data View Web Part. We will also add editing links to the Data View Web Part. The next procedure shows you how to add a Data View, as a multiple item view with editing links, to a SharePoint page.

In SharePoint Designer, go to a page and place the cursor on the page where you want to place your Data View. In this example, we will use the default.aspx page of a test SharePoint site.
Click Data View on the menu bar and select Insert Data View. This adds an empty Data View Web Part on the page and opens the Data Source Library task pane, if it is not already open.
In the Data Source Library task pane, expand the SharePoint Lists section.
Click the Announcements data source and select Show Data. This opens the Data Source Details task pane.
Hold the CTRL key down and select the Title and Expires fields. From the Insert Selected Fields as drop-down list, click Multiple Item View. Note that the order you click the fields is the order the fields will appear in the page.

The Data View Web Part is added to the page and the Common Data View Tasks action panel opens.
Click Data View Properties to open the Data View Properties dialog box.
Select the Editing tab and select the following checkboxes.
- Show edit item links
- Show delete item links
- Show insert item link
Click OK.
Save the SharePoint page in SharePoint Designer. If this is the first time you have saved default.aspx, a Warning message is displayed, stating that saving your changes will result in customizing the page. Click Yes.
Open it in a browser. The figure below shows a Data View Web Part with editing links, after clicking the edit link of the first record.

You can not only use the Data View Web Part to create data overviews, as its name implies, you can also add create, update, and delete functionality.

Creating Item Forms

Using the Item forms option of a Data View, are ideal for collecting data from users. Using SharePoint Designer, you can insert a Data View as a form so that users can make and save changes to the data source. Not all data sources can use a Item Form to insert data, the following types of data sources are supported.

Lists and libraries.
Database connections.
Local XML files.

When inserting a data view as an Item Form you can choose between three different types of forms.

Single Item Form. This form displays a single record as a form, each field will have its own row and the field name will be the heading for that row. This way it is easy to edit the data for on that record. SharePoint Designer provides navigational links, which you can use to move on to the next or previous record.
Multiple Item Form. This form displays multiple records as a form. Here you can edit the data for multiple records and save all changes at the same time.
New Item Form. This form displays blank fields for a single new record. This way you can add one new record to the data source.

In the next example, we are going to use the SharePoint Announcements list as a data source for our Data View Web Part. We will add the fields as a single item form to the Data View Web Part. Follow the next procedure to add a data view as a single item form to a SharePoint page.

In SharePoint Designer, go to a page where you want to place your Data View. In this example, we will use the default.aspx page of a test SharePoint site.
Click Data View on the menu bar and select Insert Data View. The Data Source Library task pane opens, if it is not already open.
In the Data Source Library task pane, expand the SharePoint Lists section. Here you will find the Announcements data source.
Click Announcements and select Show Data. This opens the Data Source Details task pane.
Hold down the CTRL key, and select the Title, Created and Expires fields and then from the Insert Selected Fields as drop-down list select, Single Item Form.
Save the SharePoint page in SharePoint Designer and then open the SharePoint page in a browser. Click the record arrow to navigate to the data you wish to amend and then click Save. The figure below shows a Data View Web Part inserted as a single item form with record 2 displayed.

Forms allow fine-grained control over the look and feel of each field. You can add create, read, update, and delete functionality for multiple items at a time. To enhance the user interaction experience, it is often preferable to create a form that can manipulate a single item at a time. Providing such functionality, is well within the reach of the Data View Web Part.

Working with Related/Linked Data Sources

Organizations often store data in different types of repositories, for instance, XML Files and relational databases. With the Linked Source data source, it is possible to relate different types of data sources to each other. This way, you can combine several types of data sources into a single data source that can be used by the Data View Web Part. In this section, you will learn how to create new types of data sources that combine the information stored in other data sources. Such data sources are also known as linked data sources, during the course of this article you will find out that working with a linked data sources is identical to working with any other data source.

Merging data sources

By creating a linked data source, you can merge together data from separate data sources. Such data sources need to be of the same type and they need to have a field in common that you can use to merge the information together. The WSS SDK mentions that the set of fields in both data sources have to match exactly, but this is not entirely true, as you will see later in this section.

You can only merge data sources if they are of the same type. For instance, you can merge two XML file data sources, but you cannot merge an XML file data source with a SharePoint list data source. However, you can join data sources of different types together. We discuss joining data sources in the next section.

Merging data sources allows you to create a single data source that consists of several other data sources. This single data source is a union of the other data sources.

The next example was created with one of our customers in mind, a telecom company. They knew we were working on this book and during our consulting work; they repeatedly tried to come up with cool ideas to write about. So, guys, thanks for working together, the next example is for you. In this example, we are going to create a linked data source that consists of two data sources that we will merge. The first data source is a SharePoint list called Clients. An imaginary phone company maintains this list, which contains two columns: ClientName and Minutes. The ClientName column contains the name of the customer, the Minutes column relates to the number of mobile minutes bought by the customer. The second data source is also a SharePoint list (remember, when merging data sources they have to be of the same type) called Customer. The Customer list has two columns, CustomerName and MobileMinutes. The customerName column contains the name of the customer and is really the same as the ClientName column of the first list.

Now our customer wanted a page that displayed a list of both it’s clients and customer. Follow the next procedure to link these two data sources together in one data source and show them in a Data View Web Part.

Open SharePoint Designer and open the SharePoint page where you want to place the Data View Web Part.
On the Data View menu select Insert Data View.
Go to the Data Source Library task pane and expand the Linked Sources section and click Create a new Linked Source. This opens the Data Source Properties dialog box.
Click the General tab and enter a name for the data source, for example, Customer Data.
Click the Source tab and click Configure Linked Source. This opens the Link Data Sources Wizard window.
On the left, under Available Data Sources, all available data sources are listed. Expand SharePoint Lists, select Customer and click Add. Select Clients and click Add. If the Data Source Library does not display a list that was made recently, you may have to click the Refresh Library link in the Data Source Library task pane.

Click Next. The second page of the Link Data Sources Wizard is displayed.
Select the radio button, Merge the contents of the data sources, which is the default and click Finish.
Click OK to close the Data Source Properties dialog box.
In the Data Source Library task pane, you will see a new data source under the section Linked Sources.

The figure below shows a visual impression of merging data in multiple data sources. As you can see, the linked data source contains the union of multiple other data sources.

Joining data sources

It is possible to create a joined data source that consists of multiple separate data sources, which do not have to be of the same type. The data sources that you join must have at least one field in common. Since we absolutely love to watch movies, we will show how to join data sources using some data about movies we have just created. In the example discussed in this section, we created an XML file called Genres.xml, which contains movie genres and looks as follows.

<?xml version="1.0" encoding="utf-8" ? >
<Genres>
  <Genre>
    <Id>1</Id>
    <Name>Romance</Name>
    <Description> Romantic feel good movies.</Description>
  </Genre>
   <Genre>
    <Id>2</Id>
    <Name>Horror</Name>
    <Description>Scary stuff.</Description>
   </Genre>
  <Genre>
    <Id>3</Id>
    <Name>Science Fiction</Name>
     <Description> Futuristic adventures in space. </Description>
  </Genre>
</Genres>

We called the second XML file, Movies.xml, which contains a list of movies and a genre id that matches the id of the genres listed in Genres.xml. Movies.xml looks like this.

<?xml version="1.0" encoding="utf-8" ?>
<Movies>
  <Movie>
     <Name>Fallen</Name>
     <GenreId>2</GenreId>
     <LeadingActors> Denzel Washington, John Goodman </LeadingActors>
  </Movie>
  <Movie>
     <Name>Music and Lyrics</Name>
     <GenreId>1</GenreId>
     <LeadingActors> Hugh Grant, Drew Barrymore </LeadingActors>
  </Movie>
  <Movie>
     <Name>Intolerable Cruelty</Name>
     <GenreId>1</GenreId>
     <LeadingActors> George Clooney, Catherine Zeta-Jones </LeadingActors>
  </Movie>
</Movies>

Create the two XML files with the content detailed above and store them in a document library, or use the procedure explained earlier in this article, “Adding an XML File as a data source”. We will use the element of the Genres.xml file to join it with the column of the Movies.xml file and thereby link both data sources together. The next procedure shows you how to create a joined linked source.

Open SharePoint Designer and open the SharePoint page where you want to place the Data View Web Part.
In the Data Source Library task pane, click Refresh Library and check that the two XML files appear in the XML Files section.
On the Data View menu select Insert Data View.
In the Data Source Library task pane expand the Linked Sources section and click Create a new Linked Source. This opens the Data Source Properties dialog box.
Click the General tab and specify a name for the data source, for example, Movie Data.
Click the Source tab and click Configure Linked Source. This opens the Link Data Sources Wizard window.
Under Available Data Sources., select Genres.xml file from the XML Files section and click Add. Select Movies.xml file and click Add.
Click Finish and then click OK.

In the next procedure, we show you how to add the linked data source to a SharePoint page and how to make a subview within a data view.

In the Data Source Library task pane, click the data source you just created, Movie Data, and then click Show Data.
First, we are going to create a Data View of the Genres data source. In the Data Source Details task pane, under Genre, hold down the CTRL key, click Name and Description and then click Multiple Item View from the Insert Selected Fields as menu to insert the selected data into the Data View.
The next thing we are going to do is to create a new column to show our subset of data. Right-click in the cell with the name Description, select Insert and then select Columns to the Right.
Place your cursor in a cell of the new column. Do not place the cursor in the heading cell, that is do not place the cursor in the same row as the word Description.
Then in the Data Source Details task pane, hold down the CTRL key and select Name and LeadingActors fields from the Movie folder. Click Insert selected fields as > Joined Subview. This opens the Join Subview dialog box. In this example, we are creating a subview of the movies.xml data source that is joined to the Genres.xml data source by the field genre id, which both data sources have in common. This is called a joined subview.
In the Join Subview dialog box, select the genre id fields of both data sources. In the Genres.xml file select the field called Id, and in the Movies.xml file select GenreId.
Click OK. The figure below shows the Data View Web Part showing the end result.

Using XPath queries within the Data View

One of the new features of SharePoint Designer 2007 is called the XPath Expression Builder, which makes is easy to use the XML Path Language (XPath) . XPath is a querying language for XML and is designed to be used in combination with the Extensible Stylesheet Language Transformations (XSLT) . XSLT is designed to transform XML documents into other documents.

You can use the XPath Expression Builder in SharePoint Designer to create a formula column. A formula column displays the results of a calculation that was performed on other data in a data source. The XPath Expression Builder can be used to help build expressions, or calculations that other options of the Data View Web Part can use, for example using our previous example, you may wish to conditionally format the data in the Minutes column if the number of minutes a customer uses goes above 10,000. This is a very simple example, and you could use the default Condition Criteria dialog box to create this, however under the covers this dialog box is creating XPath expressions. The XPath Expression Builder allows you to build complex expression, that the default dialog boxes cannot create for you and offers easy access to the names of the fields in the data source, as well as to many of the built-in functions that are available when writing XPath expressions.

Using the XPath Expression Builder

The XPath Expression Builder makes it very easy to create XPath expressions and even provides built-in IntelliSense for XPath. You can use the XPath Expression Builder to create the following.

Complex sort expressions
Complex filters
Conditional expressions
Formula columns

In the rest of this section we will describe how to build the first three types, and then the next section we will detail how to use the XPath Expression Builder with formula columns.

You can use the XPath Expression Builder to create XPath expressions that perform complex sort orders on data, as we show in the next procedure.

Click on the chevron arrow in the upper right corner of the Data View to open the Common Data View Tasks action panel.
Click on the Sort and Group option to open the Sort and Group dialog box.
Select Add Sort Expression, which is the last field in the Available fields list box and click Add. This opens the Advanced Sort dialog box where you can create XPath expressions. When you are finished using the Advanced Sort dialog box, click OK twice.

Next we will use the XPath Expression builder to create advanced filters as follows.

Click on the chevron arrow in the upper right corner of the Data View to open the Common Data View Tasks action panel.
Click Filter, this will open the Filter Criteria dialog box.
Select the checkbox Add XSLT Filtering; this enables the Edit button.
Click Edit to open the Advanced Condition dialog box where you can create XPath expressions. When you are finished using the Advanced Condition dialog box, click OK twice.

You can perform XPath queries on all types of data sources, even when the data in a data source is not natively stored as XML, for example, the data in a SharePoint list or SQL database. This is possible because such data is first rendered as XML by the Data View infrastructure, before the XPath expressions are applied to the data. You should also take note that XPath sorting and filtering is performed on the entire data set that has been retrieved from the data source and loaded into memory. Therefore, for performance reasons, it may be best to define your sorting and filtering criteria on the data source than defining it on the Data View.

You can use the XPath Expression Builder to build expressions that specify conditions for when applying conditional formatting as follows.

Click on the chevron arrow in the upper right corner of the Data View to open the Common Data View Tasks action panel.
Click Conditional Formatting; this opens the Conditional Formatting task pane. The Conditional Formatting task pane shows all the existing conditions in the selected Data View.
Click on the arrow next to an existing condition and choose Edit condition, this opens the Condition Criteria dialog box.
Click the Advanced button to open the Advanced Condition dialog box that lets you create XPath expressions.

The next figure shows the XPath Expression Builder, in this particular instance the Advance Condition dialog box. The text field on the left, right below the caption Select a field to insert, shows a hierarchical representation of the data structure of the selected data source.

These so-called fields can be used within an XPath expression. If you double-click a field it is added to the upper right text box just below the Edit the XPath expression caption. Alternatively, you can drag and drop field to the Edit the XPath expression text field.

The Edit the XPath expression text field is intended to create the XPath expression you need and to make this as easy as possible. The Preview section of the screen, at the bottom, provides an up-to-date representation of live data after applying an XPath expression to that data. If you change something in the Edit the XPath expression text field this change is reflected immediately in the Preview section.

Another really nice feature of the XPath Expression Builder is that the Edit the XPath expression text field displays an IntelliSense list that contains all available XPath operators as well as the fields that are available within a given context.

The Select a function category drop down list offers the choice between different types of categories of XPath functions, like Most Recently Used, All, and Math/Number. If you select a category, the list box below the Select a function to insert caption is updated so that it contains all XPath functions that belong to the selected category. The XPath functions can also be used within XPath expressions.

We discussed how using the XPath Expression Builder we can create complex sort expressions, complex filters and conditional expressions. Next, we detail how to create formula columns.

Creating a formula column

In this section, we will demonstrate how to create a formula column based on information found in different elements in an XML data source. Imagine, if you will, a mobile phone company that keeps track of the mobile minutes a customer uses and the price of one mobile minute. This information is stored in an XML file. The company knows how many mobile minutes are used, and the price of a mobile minute, but they want to create a Data View to display the total price of all mobile minutes used by customer.

In this example, you will create a formula column that displays the result of multiplying the mobile minutes by the price. When the formula column is created the mobile phone company can quickly and easily access the total amount of money they get from their customers.

We have created a sample XML file containing the mobile minutes and the price per minute per customer. If you want to follow this example, you should create this file yourself. The contents of the XML file called customers.xml are as follows.

<?xml version="1.0" encoding="utf-8"?>
<Customers>
  <Customer Name="A">
    <MobileMinute>5000</MobileMinute>
    <PricePerMinute>0.25</PricePerMinute>
  </Customer>
  <Customer Name="B">
    <MobileMinute>10000</MobileMinute>
    <PricePerMinute>0.20</PricePerMinute>
  </Customer>
  <Customer Name="C">
     <MobileMinute>15000</MobileMinute>
     <PricePerMinute>0.15</PricePerMinute>
  </Customer>
</Customers>

First, you have to add the customers.xml file as a data source. In section “Adding an XML File as a data source”, we have explained the steps you have to follow to import an XML file data source. The next procedure shows you how to create a Data View and add a formula column.

In SharePoint Designer, go to a page where you want to add your Data View. In this example, this will be the default.aspx page of a test SharePoint site.
Click Data View on the menu bar and select Insert Data View.
Click customers.xml in the XML Files section of the Data Source Library task pane and select Show Data. This opens the Data Source Details task pane.
Hold the CTRL key down, select Name, MobileMinute and PricePerMinute fields and then select Insert Selected Fields as > Multiple Item View.
On the Data View Web Part, click the chevron in the upper right corner of the Data View to open the Common Data View Tasks action panel.
Click Edit Columns. This opens the Edit Columns dialog box.
Select Add Formula Column in the Available Fields list and click Add. This opens the XPath Expression Builder dialog box. This looks very similar to the Add Condition dialog box you saw in the previous section.
In the Select a field to insert box, double-click the MobileMinute field to place it in the Edit the XPath expression box. If you want to insert the full path of the field, hold down the CTRL key while you double-click or drag the field.
Place your cursor in the Edit the XPath expression box directly after the MobileMinute field and press the SpaceBar. This displays an IntelliSense list containing all available XPath operators. IntelliSense for XPath helps creating XPath expressions by providing a list of available fields or functions that are valid in the context of the expression. Double click the * operator and then double click PricePerMinute. The expression you have build will multiply the value contained in the MobileMinute field with the value in the PricePerMinute field.

The final XPath expression looks like this: MobileMinute * PricePerMinute. In the Preview box you can see a preview of the results of the data in the formula column based on real data in the specified data source. Click OK in the XPath Expression Builder dialog box and click OK to close the Edit Columns dialog box.
The middle of the SharePoint Designer screen shows a preview of the Data View Web Part. Here you will see a new column called MobileMinute * PricePerMinute. To change the name of column, you can select the name of the column header and type a new name, such as, Payment Due. You can alter the other column headings in a similar way, and by entering Customer to the right of A, SharePoint Designer will prefix the values in the Name field with Customer too.
Save the page and browse to the default.aspx page of your SharePoint site that contains the Data View Web Part.

You may be saying to yourself that you can do these sorts of calculation using the calculated column in a list or a library, which is true, but remember by using the Data Source Library, you can access more data than that which is stored in SharePoint lists and library. If you have used the XPath query language in the past, you probably agree that it can be tricky to use. As a general trend, we notice that all sorts of commercial tools try to abstract the complexity of using XPath away by providing visual aids. A good example of such a tool created by Microsoft is the BizTalk mapper. You can now add the Data View Web Part and SharePoint Designer to the list of tools that make it easier to work with XPath.

Summary

The power of the Data View Web Part tends to be underestimated. We are sure you will not make the same mistake now you have read this article. It has been quite a journey and we hope you have tried out the various techniques we described. If you have not, remember that it is not too late! If you find it difficult at first, remember that Pablo Picasso once said, “I am always doing what I can not do, in order that I may learn how to do it”.

The article started with an introduction of the Data View Web Part and SharePoint Designer. Both tools might be new to you, and it typically takes some time getting used to the idea that you can use the Data View Web Part to aggregate and manage data.

In order to be able to aggregate and manage data, you will first need to define which data sources you want to use and how you can access these data sources. Therefore, we looked at importing and displaying data with the Data View Web Part. You saw that the Data Source Library plays a vital role in accessing data sources and you learned the kind of data sources accessible to the Data View Web Part, such as SharePoint lists, relational databases and XML files.

After demonstrating the basics of consuming data sources, you learned about more advanced topics. You saw that it is possible to connect to data sources located in other SharePoint sites. You also learned how to filter, sort, and group information displayed in the Data View Web Part. If you are interested in building sophisticated user interfaces, you were probably happy to learn about the possibilities of changing the Data View layout, generating test data, and previewing the result.

Sometimes you will be interested in creating read-only reports, so in this article you have learned how to create a data source as a view to accomplish this. At other times, you will also be interested in editing the data displayed by the Data View Web Part and you saw how to create a form to cater for this need. The article also provided you with a nice overview of the exact differences between Views and Forms.

It was interesting to explore the possibility of how to combine separate data sources using Linked data sources. We showed examples of a merged data view, an intersection of data, and a joined data view, a union of data.

The final part of the article was dedicated to the XPath Expression Builder. The XPath query language is a great language for retrieving data from XML files. In the past, the downside of using this language has been its complexity and its syntax that is rather difficult to read. The XPath Expression Builder offers a development environment that includes IntelliSense support, making development efforts using the XPath language easier.

When all is said and done, the most amazing thing about the Data View Web Part is its capability to build advanced user interfaces in a short amount of time. We would not want to go on record, stating that building Web Parts that aggregate and manage data within a SharePoint environment is a complex mission to undertake, but the boost of productivity that is offered by the Data View Web Part is a force to be reckoned with.

The Custom configuration feature - Update

19 February 2009 07:17:13

February 19, 2009

Thanks to the sharp eyes of Tony Yin (thx Tony!) we were able to improve the custom configuration feature (aka the custom application configuration, the name under which it is known at codeplex). The custom configuration at the list level was only targeted towards document libraries (because the Location property of the <CustomAction> element was targeted towards Microsoft.SharePoint.ListEdit.DocumentLibrary). We've changed this so it targets to other types of lists as well (by targeting the Location property of the <CustomAction> element towards Microsoft.SharePoint.ListEdit). You can download version 1.1 of the custom configuration feature here, and we'll post an update to codeplex as well.

The Custom configuration feature

08 December 2008 14:15:40

The Custom configuration feature

December 8, 2008

When it comes to storing application configuration settings there are lots of options available to you. Currently, we favour storing app config settings in SharePoint itself and allow administrators to control those settings via the user interface at various levels.

If you choose to store app config settings in SharePoint two options jump out as being particularly interesting: the hierarchical object store and the various SharePoint property bags. For a discussion of the hierarchical object store versus SharePoint property bags, please refer to a previous blog post we’ve written about this topic called "Hierarchical object store vs. Property bag".

In the past, Scot Hillier has blogged about using the hierarchical object store to hold web application settings in an article called "Managing the Hierarchical Data Store" and he has taken the effort to create the Manage Hierarchical Object Store feature (to be downloaded at http://www.codeplex.com/features) which adds a nice UI that allows you to manage those web app settings.

In addition to having the ability to store app settings at the web application level we find we generally need to store app settings at other levels as well, at least at the site collection, site, list and web part level. Storing web part app settings is easy, out of the box there’s a nice way to interact with web part properties, so there’s no need to address that. To add the possibility to control site collection, site and list app settings via the user interface, we’ve written a feature that adds those capabilities.

To create this feature, which we’ll call the Custom configuration feature, we’ve taken the liberty of taking the code written by Scot Hillier from the Manage Hierarchical Object Store feature and changed it so it suits our needs. There are two major differences:
- Our custom configuration feature allows you to specify app settings at the site collection, site and list level (and not the web application level).
- Our custom configuration feature uses the site and list root folder property bags to store app settings (and not the hierarchical object store). We prefer to use property bags instead of the hierarchical object store for performance and stability reasons.

Later on, we’ll try and see if Scot Hillier wants to add the Custom configuration feature to his Features project on Codeplex, but for now you can only download the Custom configuration feature here.

Installing the Custom configuration feature is pretty simple. Just download the SharePoint solution file called Custom.Configuration.cab and store it somewhere on the server. Open a command prompt, navigate to the folder that contains this cab file and add it to the SharePoint solution store, then deploy it. To accomplish this, you need to type the following commands:
stsadm -o addsolution -filename Custom.Configuration.cab
stsadm -o deploysolution -immediate -name Custom.Configuration.cab
stsadm -o execadmsvcjobs
iisreset

At this point, SCA should display the custom configuration solution, as can be seen in the next figure:

If you want to uninstall the solution, you need to open a command prompt and type the following commands:
stsadm -o retractsolution -name Custom.Configuration.cab -immediate
stsadm -o execadmsvcjobs
stsadm -o deletesolution -name Custom.Configuration.cab -override
iisreset

After adding and deploying the solution, you need to activate a site collection feature called Custom configuration settings. You can do this via the user interface:
1. Go to the Site Settings page of a site collection.
2. Under section Site Collection Administration, click the Site collection features link. This opens the Site Collection Features page.
3. Locate the Custom configuration settings feature and click the Activate button.

This activates the feature. Alternatively, you can open a command prompt, navigate to the Features folder in the 12 hive and type the following command: stsadm -o activatefeature -name CustomConfiguration -url [URL site collection]
iisreset

You can deactivate this feature via the user interface or via the following command: stsadm -o deactivatefeature -name CustomConfiguration -url [URL site collection]
iisreset

The next figure shows the custom configuration settings site collection feature in deactivated state:

The following procedure explains how to use the Custom site config settings:
1. Open the Site settings page of any SharePoint site.
2. In the Custom site config settings page, click the Custom site config settings link. This link is shown in the next figure:

3. This opens the Custom site config settings page. This page provides all current site config settings, which is shown in the next figure:

4. If you press the New button, you’ll be taken to a page that allows you to add a new application setting at the site level. If you click on the Edit button, you‘ll be taken to a page that allows you to adjust an application setting. This is shown in the next figure:

The following procedure explains how to use the Custom list config settings:
1. Open the List settings page of any SharePoint list.
2. In the General Settings section, click the Custom list config settings link. This link is shown in the next figure:

3. This opens the Custom list config settings page. This page provides all current list config settings, which is shown in the next figure:

4. If you press the New button, you’ll be taken to a page that allows you to add a new application setting at the list level. If you click on the Edit button, you‘ll be taken to a page that allows you to adjust an application setting. Adding a new application settings is shown in the next figure:

Please note that the Custom configuration feature requires site admin rights, otherwise, you won’t be able to open the config management pages. If you don’t want this, you should comment out the following code in both .cs files that can be found in the CustomConfiguration folder of the LAYOUTS folder of the 12 hive: protected override bool RequireSiteAdministrator
{
get
{
return true;
}
}

Reading application settings is pretty easy once you have a valid reference to a SharePoint site or list. For instance, if you’re able to access the SPContext object you could use the following line to read an app setting for the current SharePoint site: SPContext.Current.Web.Properties[strKey];

And you could use the following line to read an app setting for the current SharePoint list: SPContext.Current.List.RootFolder.Properties[strKey]

If you need to access SharePoint site collection settings, just read the properties of the root web site of a SharePoint site collection.

As a side note, since all our custom application pages make use of the application.master Master page file they have inherent support for breadcrumb navigation. In our development environment we’ve just added some nodes to the layouts.sitemap file in the App_bin folder of the SharePoint web application that hosts our custom configuration feature and added the appropriate navigation info.

To add navigation info for custom site app config settings, locate the <siteMapNode title="$Resources:wss,settings_pagetitle" url="/_layouts/settings.aspx"> node and add the following nodes right below it: <siteMapNode title="Site configuration" url="/_layouts/CustomConfiguration/SiteConfig.aspx">
<siteMapNode title="Adding a site configuration setting" url="/_layouts/CustomConfiguration/AddSiteConfig.aspx" />
</siteMapNode>

To add navigation info for custom list app config settings, locate the <siteMapNode title="$Resources:wss,lstsetng_settings" url="/_layouts/listedit.aspx" requiredParameters="List"> node and add the following nodes right below it: <siteMapNode title="List configuration" url="/_layouts/CustomConfiguration/ListConfig.aspx">
<siteMapNode title="Add list configuration setting" url="/_layouts/CustomConfiguration/AddListConfig.aspx" />
</siteMapNode>

As stated before, adding navigation info like this is okay on a development environment. You’d actually be better of configuring a custom sitemap provider in the web.config file of the SharePoint web application hosting the custom configuration feature, create a new site map file and create a new master page file that refers to your custom site map. The article "Implementing Consistent Navigation across Site Collections" written by Sahil Malik discusses how to do this.

As a final word, you’re welcome to use our custom configuration feature. It’s community software and it’s free, so we’re not providing official support for it and using it is at your own risk (although the source code is provided in the solution). Having said that, if you find a bug in it, please let us know (info@lcbridge.nl) and we’ll try to fix it.

Pop quiz: what happens when indexing large files?

20 October 2008 12:38:47

Pop quiz: what happens when indexing large files?

Written by: Margriet Bruggeman, Nikander Bruggeman.

A standard MOSS installation only indexes documents up to 16Mbs. If you exceed this, the crawl logs will show a warning message (not an error message) that such a document has reached the maximum download limit (http://support.microsoft.com/kb/287231). You can change this default by setting a registry value (set HKLM\Software\Microsoft\Office Server\12.0\Search\Global\GatheringManager\MaxDownloadSize, add this key if it’s missing) to the max number of Mbs (in decimals) that you want to index.

Now, when it comes to indexing large files we’ve heard three persistent rumors:
A) Only the first 16Mb of a document is indexed, the rest is discarded (to us, this sounds as the most reasonable approach).
B) The content of a document isn’t indexed at all.
C) The document isn’t indexed at all.
D) Only the first 100K of the document is indexed (to us, this sounds as the weirdest possible answer of all four).

The pop quiz is of course which of the answers above is the correct one. Getting curious, we’ve tried it out and found that the content of a document that is larger than the maximum download size (which is either larger than the default size of 16Mb or, if present, larger than the size currently specified via the MaxDownloadSize key) doesn’t get indexed at all. However, the properties of such a document (like title and last modifier) get indexed correctly. So the correct answer is B.

Further more, we love to use the Filtdump tool to help diagnose indexing problems (http://msdn.microsoft.com/en-us/library/ms951558.aspx ). If you’re not using it already, you should check it out. We’ve noticed that this tool doesn’t seem to care about this limit and shows the entire content of a document that is too large correctly.

Finally, we’ve written another post that is kinda related to this one. It’s about working with large files in a SharePoint environment. If this interests you, you should check it out at (http://www.lcbridge.nl/vision/2008/largefiles.htm).

Create an Autocomplete Custom Field Type with Ajax for ASP.NET

15 September 2008 08:09:36

Create an Autocomplete Custom Field Type with Ajax for ASP.NET

Written by: Margriet Bruggeman, Nikander Bruggeman.

There’s been a lot of discussion about the question if MOSS 2007 should support Ajax for ASP.NET or not. We’ve heard a lot of people say that it’s nice to have, but that there are other issues that should have gotten more priority. Other people are unimpressed with Ajax for ASP.NET altogether, because they say: "This possibilities have been around for ages, I could have done that since the days of XML data islands!" And yet other people don’t like entire the concept of Ajax, claiming it doesn’t solve customer problems that are really crucial. Be that as it may, MOSS service pack 1 officially happens to support Ajax for ASP.NET and we love it and we use it. Our personal favourite is the Autocomplete behaviour and in this article we’ll show how to build a custom field type (CFT) that uses it.

What are we doing?

We had this article lying around for some time, and never got around to finishing it. We’ve finished it now, so here goes...

In this article we’ll create a custom field type that offers suggestions to the end user while he types. Before accomplishing this, we need to enable ASP.NET for Ajax support in MOSS 2007. Once that’s set up and ready to go, we’ll create a SharePoint list that holds all available suggestions which makes it easy for end users to modify the list of suggestions. After that, we’ll create a simple web service that is able to retrieve data from this list and supports ASP.NET for Ajax. This web service will act as the end point to which the CFT communicates. Next, we’ll build the CFT, add required configuration information for SharePoint and try it all out.

Enable ASP.NET for Ajax

Enabling ASP.NET for Ajax is a process that can be quite painful. We’ve done this a couple of times for SPS 2003, MOSS 2007 beta and the final version of MOSS 2007 by creating an ASP.NET for Ajax web site and taking a good look at its web.config file and comparing it with the web.config file of our SharePoint installation. We had to do this because we were in the process of writing several chapters about SharePoint and Ajax for different books. As we don’t do this often, we think it would be okay to plug those books a bit here:

Pro SharePoint 2007 Development Techniques (written in its entirety by us).
Pro SharePoint 2003 Development Techniques (written in its entirety by us).
Developer's Guide to the Windows Sharepoint Services V3 Platform (lead author: Todd Bleeker, we were contributing authors).

For more information about one of these books, check Amazon (or another online book vendor) or our web site at http://www.lcbridge.nl/publications.htm. If you want to build ASP.NET for Ajax solutions for MOSS 2007, we think our book "Pro SharePoint 2007 Development Techniques" would be a valuable addition to your library. Anyways, enough of the sales pitch.

Luckily, as far as ASP.NET for Ajax support in SharePoint goes, things have improved over time. Mike Ammerlaan of the Microsoft SharePoint team wrote an extensive blog post about the topic of Ajax-enabling MOSS that guides you through the process of enabling ASP.NET for Ajax. Although it is still tiresome to make these changes manually, the blog post does a great job of walking you through it. You can find more information at http://sharepoint.microsoft.com/blogs/mike/Lists/Posts/Post.aspx?ID=3.

And even better... On CodePlex you can find a feature that is a part of the SharePoint 2007 Features project called the Ajax.Config feature. Currently, we like to use this feature to take care of the enabling of ASP.NET for Ajax for us. This is a web application level feature that uses the Feature event receiver to update and Ajax enable the web.config file of a SharePoint web application automatically once the feature is activated. You can download it at http://www.codeplex.com/features.

We suggest you read the blog post of Mike Ammerlaan or try out the Ajax.Config feature and go ahead and enable ASP.NET for Ajax on your SharePoint machine.

Suggestions list

We’ll create a SharePoint list that holds all available suggestions that will be made to the end user. A web service, discussed in detail in section "Suggestions web service" will be used to retrieve suggestions from this list. To create this list (which we’ll call "Suggestions"), just choose a SharePoint site, create a custom list and fill its Title column with the following test data:

- Alabama
- Alaska
- Arizona
- Arkansas
- California
- Colorado
- Connecticut
- Idaho
- Illinois
- Indiana
- Iowa

This gives you some test data to play around with once everything is set up and done.

Suggestions web service

Now let’s discuss the creation of a web service that supports ASP.NET for Ajax. The example is still an old-school web service, not a WCF service. We’ve decided to do this because there are a couple of issues that need to be resolved before you’re able to use WCF in all popular usage scenarios within SharePoint. The ones we can think of on top of our head are these:

You need to adjust the web.config file again to support this (see a href="http://blogs.msdn.com/bgeorgi/archive/2007/01/06/calling-a-wcf-web-service-from-sharepoint.aspx" target="_blank">http://blogs.msdn.com/bgeorgi/archive/2007/01/06/calling-a-wcf-web-service-from-sharepoint.aspx for more information).
You need to ASP.NET for Ajax-enable the WCF service.
You need to find a way to virtualize the WCF service so that it’s able to run within a SharePoint context.

Mind you, we’re not saying these are unsolvable issues, but we didn’t have time to try and come up with solutions and we haven’t read about solutions for any of these topics either. As a result, our Suggestions web service will be an old-school web service. The next procedure explains how to create a virtual directory that will hold our Suggestion web service:

Open a command prompt and type inetmgr. This opens the Internet Information Services (IIS) Manager.
Open the [server name] (local computer) node.
Open the Web Sites node.
Locate the child node representing the SharePoint web application where you want to add the Ajax CFT later on.
Right-click this child node and choose New > Virtual Directory. This opens the Virtual Directory Creation Wizard.
Click Next. This opens the Virtual Directory Alias dialog window.
In the Alias textfield, type SuggestionHost.
Click Next. This opens the Web Site Content Directory window.
Browse to the location that will hold the Suggestion web service. We've chosen the location C:\inetpub\wwwroot\SuggestionHost ourselves.
Click Next. This opens the Virtual Directory Access Permissions window.
Accept all default settings and click Next.
This completes the Virtual Directory Creation Wizard. Click Finish.
Right-click the SuggestionHost node and choose Properties. This opens the SuggestionHost Properties window.
Click the Create button.
Click OK.

Now you've successfully created a virtual directory that'll be able to hold our Suggestion web service. Next, we'll create the service itself.

Start VS.NET 2008 (at least, that's the version we're using ourselves).
Choose File > New > Web Site. This opens the New Web Site window.
In the Templates section, choose ASP.NET Web Service.
In the Location drop down list, choose HTTP.
In the Location textfield, type: http://[your_server:your_port]/SuggestionHost.
Click OK.
Right-click the SuggestionHost project and choose Add New Item. This opens the Add New Item - http://[your_server]/SuggestionHost window.
In the Templates section, choose Web Service.
In the Name textfield, type SuggestionService.asmx.
Make sure the Place code in separate file checkbox is selected.

We've created the basics of the Suggestion web service. The SuggestionService.asmx file won't change anymore and should look like this: <%@ WebService Language="C#" CodeBehind="~/App_Code/SuggestionService.cs" Class="SuggestionService" %>

We've also created a file called SuggestionService.cs. We won't list this one yet, as it still needs a little work. The first thing that this web service needs is to retrieve all suggestions stored in the SharePoint list discussed in section “Suggestions list”. In the next code listing, we’ll demonstrate how to retrieve all items from our predefined SharePoint suggestions list. We’ll use to SharePoint object model to loop through each of the list entries and add the titles of all items to a list of strings, so we can work on that list later.

Please note: This piece of the code would be a nice place to add some caching, so you won’t have to retrieve the suggestions again from SharePoint.

This code listing shows the suggestion retrieval: private void InitItems()
{
if (_dvItems == null)
{
  DataTable dtItems = new DataTable();
  dtItems.Columns.Add("ListItem", typeof(string));

  using (SPSite objSites = new SPSite(ConfigurationManager.AppSettings["http://[my server]"]))
  {
   SPList objList;

   SPWeb objWeb = objSites.OpenWeb();
   objList = objWeb.Lists["Suggestions"]];

   SPListItemCollection objListItems = objList.Items;

   foreach (SPListItem objListItem in objListItems)
   {
    DataRow drItem = dtItems.NewRow();
    drItem["ListItem"] = objListItem.Title;
    dtItems.Rows.Add(drItem);
   }
  }

  _dvItems = dtItems.DefaultView;
  _dvItems.Sort = "listitem ASC";
}
}

We’ve taken care of the retrieval of suggestions from our SharePoint list, but we haven’t finished our web service yet. The ASP.NET Ajax auto completion web service interface demands a very strict interface that requires a given interface and doesn’t allow a change in parameter names. public List<string> MyMethodName(string prefixText, int count)

The prefixText argument contains the prefix of any match found by the autocompletion web service, the count argument limits the max number of results that are returned to the client and the return type of this method needs to be some type of string array. A generic List collection of strings is allowed, and that’s the return type we’ll use here. Luckily, there’s still one thing you’re allowed to change and that is the method name itself. We’ll use that liberty to the fullest and call our method GetItems(). In this method, we’ll use a data view to filter our results: [WebMethod]
public List<string> GetItems(string prefixText, int count)
{
string strReturn = String.Empty;

try
{
  InitItems();
  string strFilter = String.Format("listitem LIKE '{0}%'", prefixText);
  _dvItems.RowFilter = strFilter;

  List<string> lstItems = new List<string>();
  for (int i = 0; i < _dvItems.Count; i++)
  {
   if (i > count - 1) break;
   lstItems.Add(_dvItems[i]["ListItem"].ToString());
  }

  return lstItems;
}
catch (Exception err)
{
  Trace.Write(err.Message);
  throw;
}
}

This concludes our auto completion web service; the complete code for this web service looks like this: using System;
using System.Collections;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Text;
using System.Diagnostics;
using System.Threading;
using System.Web.Script.Services;
using System.Data;
using System.Collections.Generic;
using System.Configuration;
using Microsoft.SharePoint;

/// <summary>
/// Summary description for SuggestionService
/// </summary>
[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
// To allow this Web Service to be called from script, using ASP.NET AJAX, uncomment the following line.
// [System.Web.Script.Services.ScriptService]
public class SuggestionService : System.Web.Services.WebService
{
public SuggestionService ()
{
  //Uncomment the following line if using designed components
  //InitializeComponent();
}

private DataView _dvItems = null;
private string[] _arrItems = null;
private bool _blnInitComplete = false;
List<string> _objSuggestionList = null;

private void InitItems()
{
  if (_dvItems == null)
  {
   DataTable dtItems = new DataTable();
   dtItems.Columns.Add("ListItem", typeof(string));

   using (SPSite objSites = new SPSite(ConfigurationManager.AppSettings["http://[my server]"]))
   {
    SPList objList;

    SPWeb objWeb = objSites.OpenWeb();
    objList = objWeb.Lists["Suggestions"]];

    SPListItemCollection objListItems = objList.Items;

    foreach (SPListItem objListItem in objListItems)
    {
     DataRow drItem = dtItems.NewRow();
     drItem["ListItem"] = objListItem.Title;
     dtItems.Rows.Add(drItem);
    }
   }

   _dvItems = dtItems.DefaultView;
   _dvItems.Sort = "listitem ASC";
  }
}

[WebMethod]
public List<string> GetItems(string prefixText, int count)
{
  string strReturn = String.Empty;

  try
  {
   InitItems();
   string strFilter = String.Format("listitem LIKE '{0}%'", prefixText);
   _dvItems.RowFilter = strFilter;

   List<string> lstItems = new List<string>();
   for (int i = 0; i < _dvItems.Count; i++)
   {
    if (i > count - 1) break;
    lstItems.Add(_dvItems[i]["ListItem"].ToString());
   }

   return lstItems;
  }
  catch (Exception err)
  {
   Trace.Write(err.Message);
   throw;
  }
}
}

Since we’ve already ASP.NET for Ajax-enabled MOSS, we can’t register that stuff again in our web service web.config file. Because of this, if you’re adding this web service to a MOSS virtual server (as opposed to adding a web service to a non-MOSS virtual server), you need to comment the contents of the <httpModules>, <modules> and <handlers> elements. After that, the auto completion web service should work fine. You can download the code for the web.config file of our own auto completion web service here.

Alternatively, we could have implemented this web service using Linq. In that case you’ll be using .NET 3.5 and you’ll have to readjust the web.config file again to Ajax-enable it. We didn’t want to go to the trouble of doing that, so we can’t include that web.config file in this article. However, we did think it was fun to rewrite the service using Linq, which is shown in the next code listing:

Please note: If you want to use the Linq implementation instead of the data view implementation, you’ll have to change the web.config file yourself!!! using System;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Diagnostics;
using System.Threading;
using System.Web.Script.Services;
using System.Data;
using System.Collections.Generic;
using System.Configuration;
using Microsoft.SharePoint;
using System.Linq;
using System.Xml.Linq;

/// <summary>
/// Summary description for KeywordService
/// </summary>
[WebService(Namespace = "http://loisandclark.suggestionservice/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[ScriptService]
public class SuggestionService : System.Web.Services.WebService
{
private DataView _dvItems = null;
private string[] _arrItems = null;
private bool _blnInitComplete = false;
List<string> _objSuggestionList = null;

private void InitItems()
{
  if (_blnInitComplete) return;

  _objSuggestionList = new List<string>();

  if (_dvItems == null)
  {
   DataTable dtItems = new DataTable();
   dtItems.Columns.Add("ListItem", typeof(string));

   using (SPSite objSites = new SPSite("http://myserver"))
   {
    SPWeb objWeb = objSites.OpenWeb();
    SPList objList = objWeb.Lists["Suggestions"];
    SPListItemCollection objListItems = objList.Items;

    foreach (SPListItem objListItem in objListItems)
    {
     _objSuggestionList.Add(objListItem.Title);
    }
   }

   _blnInitComplete = true;
  }
}

[WebMethod]
public List<string> GetItems(string prefixText, int count)
{
  try
  {
   InitItems();

   IEnumerable items = (from String name in _objSuggestionList
   where name.ToLower().StartsWith(prefixText.ToLower())
   orderby name
   select name).Take(count);

   return new List<string>(items);
  }
  catch (Exception err)
  {
   Trace.Write(err.Message);
   throw;
  }
}
}

Creating a control

In this section, we’ll show a user control that can be used as a SharePoint Custom Field Type (CFT). The user control is responsible for creating the user interface of the CFT. We’ll include a ScriptManager control in it, that is required for ASP.NET for Ajax.

Please note: Since you can only add a single ScriptManager on a page, and since the control already contains one, using this implementation, you can’t add another (or two instances of the same) ASP.NET for Ajax-enabled control to a set of metadata. If you want to support multiple ASP.NET for Ajax-enabled controls, you need to make sure the ScriptManager is only added once to the page. <%@Control Language="C#" Debug="true" %>
<%@Assembly Name="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@Register TagPrefix="SharePoint" Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=
71e9bce111e9429c" namespace="Microsoft.SharePoint.WebControls"%>
<%@Register TagPrefix="ajaxToolkit" Assembly="AjaxControlToolkit" Namespace="AjaxControlToolkit" %>
<SharePoint:RenderingTemplate ID="KeywordEditRenderingTemplate" runat="server">
<Template>
  <ajaxToolkit:ToolkitScriptManager runat="server" ID="ScriptManager1" />
  <asp:TextBox ID="txtKeyword1" MaxLength="100" Width="385"
  runat="server"/>
  <ajaxToolkit:AutoCompleteExtender
  runat="server"
  ID="autoComplete1"
  TargetControlID="txtKeyword1"
  ServicePath="http://[your server]/SuggestionHost/SuggestService.asmx"
  ServiceMethod="GetItems"
  MinimumPrefixLength="1"
  CompletionInterval="1000"
  EnableCaching="true"
  CompletionSetCount="5">
  </ajaxToolkit:AutoCompleteExtender>
</Template>
<SharePoint:RenderingTemplate>

You need to copy SuggestionControl.ascx to the 12 hive TEMPLATE\CONTROLTEMPLATES folder, by default located at:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\ControlTemplates

Okay, that takes care of the UI for the CFT. You’ll also need to create an assembly that implements functionality required by the CFT. This assembly needs to be strong named because it must be deployed in the GAC and it needs to contain a class that represents the CFT. Such a class needs to inherit from base classes called SPField* located in the Microsoft.SharePoint namespace. The base class you choose depends on the functionality you need. For example, you could inherit from base classes like SPFieldAttachments, SPFieldChoice, SPFieldLink, and SPFieldDateTime. The following code shows an example implementation: using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.SharePoint;
using Microsoft.SharePoint.WebControls;

namespace CustomFieldTypeSuggestion
{
/// <summary>
/// Custom field type for keywords.
/// Suggestion CFT's offer Ajax-enabled assistance while the end user types in the value of a property
/// of a document library item.
/// The values provided by the assistance mechanism are stored in a SharePoint list.
/// </summary>
public class SuggestionField : SPFieldText
{
#region ctors
  /// <summary>
  /// Inits object
  /// </summary>
  public SuggestionField(SPFieldCollection fields, string fieldName)
  : base(fields, fieldName)
  { }

  /// <summary>
  /// Inits object
  /// </summary>
  public SuggestionField(SPFieldCollection fields, string typeName, string displayName)
  : base(fields, typeName, displayName)
  { }
#endregion

  /// <summary>
  /// Returns a user control as the field control's UI.
  /// </summary>
  public override BaseFieldControl FieldRenderingControl
  {
   get
   {
    return new SuggestionFieldControl();
   }
  }

  /// <summary>
  /// Returns the field value object.
  /// </summary>
  public override object GetFieldValue(string value)
  {
   if (!string.IsNullOrEmpty(value))
   {
    return new SuggestionFieldValue(value);
   }
   else
   {
    return null;
   }
  }
}
}

As you may have noticed, this class uses a web control called SuggestionField control which is added to the SharePoint UI. The implementation contains a text box and looks like this: using System;
using System.Collections.Generic;
using System.Text;
using System.Web.UI.WebControls;
using Microsoft.SharePoint.WebControls;

namespace CustomFieldTypeSuggestion
{
public class SuggestionFieldControl : BaseFieldControl
{
  protected TextBox txtKeyword1 = null;
  protected SuggestionFieldValue fieldValue = null;

  /// <summary>
  /// Returns the name of the ASCX template to use.
  /// </summary>
  protected override string DefaultTemplateName
  {
   get { return "KeywordEditRenderingTemplate"; }
  }

  /// <summary>
  /// Uses Field Value object to compress and extract
  /// the various decomposible elements of the keyword CFT.
  /// Every keyword CFT contains 5 keywords.
  /// </summary>
  public override object Value
  {
   get
   {
    EnsureChildControls();
    fieldValue = new SuggestionFieldValue();
    fieldValue.Keyword1 = txtKeyword1.Text.Trim();

    return fieldValue;
   }
   set
   {
    EnsureChildControls();
    fieldValue = value as SuggestionFieldValue;
    txtKeyword1.Text = fieldValue.Keyword1;
   }
  }

  /// <summary>
  /// Adds all required child controls to the keyword CFT.
  /// </summary>
  protected override void CreateChildControls()
  {
   base.CreateChildControls();

   // If the FieldMetadata Field has been created
   if (Field != null)
   {
    switch (ControlMode)
    {
     case SPControlMode.Display:
      // Display mode is handled by a CAML RenderPattern
      break;

     case SPControlMode.New:
      AssociateControlsFromUserControl(TemplateContainer);
      break;

     default:
      AssociateControlsFromUserControl(TemplateContainer);
      break;
    }
   }
  }

  /// <summary>
  /// Get each textbox from the User Control and
  /// associate it with the control's variable
  /// </summary>
  private void AssociateControlsFromUserControl(TemplateContainer usercontrol)
  {
   txtKeyword1 = usercontrol.FindControl("txtKeyword1") as TextBox;
   txtKeyword1.CssClass = this.CssClass;
  }
}
}

There’s also a class called SuggestionFieldValue that represents the values of the CFT.

Please note: Originally, we planned to create a CFT that would contain multiple values. Because of that, it inherits from SPFieldMultiColumnValue.

The next code listing shows the implementation of the SuggestionFieldValue class: using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.SharePoint;

namespace CustomFieldTypeSuggestion
{
public class SuggestionFieldValue : SPFieldMultiColumnValue
{
  private const int numberOfFields = 1;

#region ctors
  /// <summary>
  /// Inits object
  /// </summary>
  public SuggestionFieldValue(): base(numberOfFields)
  { }

  /// <summary>
  /// Adds new field in the SPFieldCollection
  /// </summary>
  public SuggestionFieldValue(string value) : base(value)
  { }
#endregion

  /// <summary>
  /// Retrieves value of the first keyword of the keyword CFT. Every keyword CFT contains a total of 5 keywords.
  /// </summary>
  public string Keyword1
  {
   get { return this[0]; }
   set { this[0] = value; }
  }
}
}

Now, you need to compile this assembly, add it to the GAC and make sure the System.Web.Extensions and AjaxControlKit assemblies are also present in the GAC.

fldtypes_Suggestion.xml

You must use CAML to define where SharePoint can find the CFT. You can define CFTs at different scopes: list level or farm level. If you define a CFT at the list level, you need to adjust the schema.xml file of that list. If you’re defining a CFT at the farm level, you’ll need to create a file starting with "fldTypes_*" and add it to the TEMPLATE\XML folder of the 12 hive.

In this article, we’ve created a file called fldtypes_suggestion.xml that defines a farm level CFT. The next code listing shows its implementation: <?xml version="1.0" encoding="utf-8" ?>
<FieldTypes>
<FieldType>
  <Field Name="TypeName">Suggestion</Field>
  <Field Name="ParentType">MultiColumn</Field>
  <Field Name="TypeDisplayName">Suggestion display name</Field>
  <Field Name="UserCreatable">TRUE</Field>
  <Field Name="Sortable">TRUE</Field>
  <Field Name="Filterable">TRUE</Field>
  <Field Name="FieldTypeClass">
   CustomFieldTypeSuggestion.SuggestionField,CustomFieldTypeSuggestion,
   Version=1.0.0.0,Culture=neutral,PublicKeyToken=1b824d71fdeef6b0
  </Field>
  <RenderPattern Name="DisplayPattern">
   <Switch>
    <Expr>
     <Column />
    </Expr>
    <Case Value="">
    </Case>
    <Default>
     <Column SubColumnNumber="0" HTMLEncode="TRUE"/>
     <HTML><![CDATA[<br />]]></HTML>
    </Default>
   </Switch>
  </RenderPattern>
</FieldType>
</FieldTypes>

You need to copy fldtypes_suggestion.xml to TEMPLATE\XML folder of the 12 hive, by default located at:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\XML

Demo

Finally, this leaves us with the end result. After following the steps in this article, you can go to a list and create a new column of the type "Suggestion display name". You’ll get a warning stating that you need to be aware that your own CFT cannot be edited from most client programs and might block the programs from saving documents to this library. So, that’s a limitation that you need to consider.

After this, you’re done. When you start editing list items, you’ll get Ajax-enabled typing support. For instance, if we start typing "i", our Suggestions web service comes up with several suggestions, as can be seen in the following screen shot:

If we type further, and type "in" we’ve only added a single state to our list that starts with those letters, "Indiana". As a result, that’s the only suggestion offered by our Suggestion web service, as can be seen in the following screenshot:

So there you have it, an AutoComplete Custom Field Type with Ajax for ASP.NET.

Hierarchical object store vs Property bag

29 August 2008 08:09:22

Hierarchical object store vs Property bag

Written by: Margriet Bruggeman, Nikander Bruggeman.

We find, when building MOSS solutions, that we generally have a need for configuration information stores at four different levels:

Web application level
Site collection level
Site level
List level

There are lots of options when it comes to choosing configuration information stores, but at the web application level SharePoint offers two choices that are quite natural: the <AppSettings> section of the web application web.config file and the hierarchical object store.

Of these two options, the hierarchical object store is by far the lesser known of the two. To put it shortly, the hierarchical object store offers a framework that allows third party applications to store configuration information by creating a class that inherits from the SPPersistedObject class in the Microsoft.SharePoint.Administration namespace. You can find more information about it here: http://www.bluedoglimited.com/SharePointThoughts/ViewPost.aspx?ID=271.

You should also take a look at http://www.codeplex.com/features, which contains a feature called "Manage Hierarchical Object Store" that offers a user interface for managing web application configuration information that is ultimately stored in the hierarchical object store. This example would be quite easy to adjust so that it caters the management of configuration info at different levels as well.

If you look at the site and list level, besides the hierarchical object store, you have another option offered by SharePoint: the property bag. At the site level, it can be accessed via the Properties property of an instance of the SPWeb class. Instances of SPList classes don’t have a property bag associated to it. However, list items do have such a property bag. Therefore, using the property bag of the root folder of a list is a natural alternative if you want to store list level configuration information.

Since we were curious how the performance of the hierarchical object store and the property bag compare to each other, we’ve devised a simple performance test incorporated in a console application. The console application stores and retrieves config info at the list level and uses three different methods for storing the data:

Storage in a single property in the property bag using XML serialization
Storage in the hierarchical object store
Storage in multiple properties in the property bag

We’re sharing the results of this simple test with you in this article.

Creating a persistable object

First, we’re creating a class that contains 10 fields that need to be persisted in the hierarchical object store. In order to do so, we’re creating a class that inherits from the SPPersistedObject class. The class will contain two constructors; one that is required by the SPPersistedObject and an empty constructor that needs to be there for serialization purposes. Every field that needs to be persisted needs to be decorated with the [Persisted] attribute. The end result is the following class: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration;

namespace HObjectStore
{
public class PersistStore : SPPersistedObject
{
  [Persisted]
  public int number1 = 1;
  [Persisted]
  public int number2 = 1;
  [Persisted]
  public int number3 = 1;
  [Persisted]
  public int number4 = 1;
  [Persisted]
  public int number5 = 1;
  [Persisted]
  public int number6 = 1;
  [Persisted]
  public int number7 = 1;
  [Persisted]
  public int number8 = 1;
  [Persisted]
  public int number9 = 1;
  [Persisted]
  public int number10 = 1;

#region ctor
  //Exists for serialization purposes.
  public PersistStore ()
  {
  }

  public PersistStore(string strName, SPPersistedObject objParent, Guid objGuid)
  : base(strName, objParent, objGuid)
  {
  }
#endregion
}
}

This class is quite simple to build, and quite simple to use as well. You can create a new instance of our PersistStore class by passing a key, a parent that also inherits from SPPersistedObject (which seems to hold true for many classes in the Microsoft.SharePoint.Administration namespace, in this example we’ll use an SPWebApplication object) and a GUID. Please note that the GUID needs to be unique, you can’t reuse the same GUID and use a different key. So, instantiating a PersistStore object looks something like this (where objSite is an instance of an SPSite object): PersistStore objTest2 = new PersistStore("MyKey", objSite.WebApplication, objList.ID);

If you’ve already persisted info, you need to retrieve it from the hierarchical object store via its key, like so: PersistStore objTest2 = objSite.WebApplication.GetChild("MyKey");

If you want to persist a persistable object in the hierarchical object store, all you need to do is call the Update() method (defined in the SPPersistedObject class), like so: objTest2.Update();

Finally, you can delete persisted info by calling the Delete() and Unprovision() methods, also defined in the SPPersistedObject class. The code for doing this is shown here: objTest2.Delete();
objTest2.Unprovision();

As you’ve seen, interacting with the hierarchical object store isn’t hard to do. We’ll be revisiting this code later in our performance test.

XML Serializing info in a single property of the property bag

We’ll also be building a class that we’ll XML serialize and store in a property of the RootFolder property bag of a list. The class itself, which we’ll call SerializeClass, is very simple. It contains 10 fields that’ll be persisted: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace HObjectStore
{
public class SerializeClass
{
  public int number1 = 1;
  public int number2 = 1;
  public int number3 = 1;
  public int number4 = 1;
  public int number5 = 1;
  public int number6 = 1;
  public int number7 = 1;
  public int number8 = 1;
  public int number9 = 1;
  public int number10 = 1;
}
}

We’ve found some code that’s able to handle the XML serialization at http://www.dotnetjohn.com/articles.aspx?articleid=173, which saves us the trouble of explaining it. We won’t list the code here, but will include it in our test code. For now, it suffices to know that this solution is more complex and requires more code. If you want specifics about this topic, please refer to the aforementioned article.

Storing info in multiple properties of the property bag

Storing info in multiple properties of the property bag is the easiest scenario of the three. Our test application wants to store list specific configuration information. Since instances of the SPList class don’t have a property bag, we’ll use the property bag of the rootfolder instead. The following code shows how to read a property that contains an integer from the root folder property bag: SPFolder objFolder = objList.RootFolder;
if (objFolder.Properties.Contains("number9"))
{
int9 = Convert.ToInt32(objFolder.Properties["number9"]);
int9++;
}

The following code shows how to add a value in the root folder property bag: objFolder.Properties["number10"] = int10;
objFolder.Update();

So, that’s that, and we’ll see this code again in the "Creating a performance test" section.

Creating a performance test

Our performance test executes tests for all three scenarios we’ve described. The first test does XML serialization, the second test uses the hierarchical object store and the third test uses multiple properties to store single values. If you’ve read the article so far, you should be able to understand the code. If not, just skip this section and look at the "Results" section. using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Xml;
using System.Xml.Serialization;
using System.Diagnostics;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration;

namespace HObjectStore
{
class Program
{
  static void Main(string[] args)
  {
   try
   {
    Stopwatch objTimer1 = new Stopwatch();
    Stopwatch objTimer2 = new Stopwatch();
    Stopwatch objTimer3 = new Stopwatch();

    int intRuns = 20;
    string TEST_KEY = "TestAppSetting";
    Guid objParentId = new Guid();
    Guid objListId = new Guid();

    using (SPSite objSite = new SPSite("[your server]"))
    {
     SPWeb objWeb = objSite.OpenWeb();
     SPList objList = objWeb.Lists["Suggestions"];

     objTimer1.Start();
     for (int i = 0; i < intRuns; i++)
     {
      SerializeClass objTest1;

      string strXmlSerializedObject = null;
      if (objList.RootFolder.Properties.Contains(TEST_KEY))
      {
       strXmlSerializedObject = objList.RootFolder.Properties[TEST_KEY].ToString();
      }

      if (strXmlSerializedObject == null)
      {
       objTest1 = new SerializeClass();
      }
      else
      {
       objTest1 = (SerializeClass)DeserializeObject(strXmlSerializedObject);
      }

      //Change object
      objTest1.number1++;
      objTest1.number2++;
      objTest1.number3++;
      objTest1.number4++;
      objTest1.number5++;
      objTest1.number6++;
      objTest1.number7++;
      objTest1.number8++;
      objTest1.number9++;
      objTest1.number10++;

      //Serialize object.
      String XmlizedString = null;
      MemoryStream memoryStream = new MemoryStream();
      XmlSerializer xs = new XmlSerializer(typeof(SerializeClass));
      XmlTextWriter xmlTextWriter = new XmlTextWriter(memoryStream, Encoding.UTF8);
      xs.Serialize(xmlTextWriter, objTest1);
      memoryStream = (MemoryStream)xmlTextWriter.BaseStream;
      XmlizedString = UTF8ByteArrayToString(memoryStream.ToArray());

      // Save serialized object.
      objList.RootFolder.Properties[TEST_KEY] = XmlizedString;
      objList.RootFolder.Update();

     }      objTimer1.Stop();

     objParentId = objSite.WebApplication.Id;
     objListId = objList.ID;

     objTimer2.Start();
     for (int i = 0; i < intRuns; i++)
     {
      PersistStore objTest2 = objSite.WebApplication.GetChild(TEST_KEY);

      if (objTest2 == null)
      {
       objTest2 = new PersistStore(TEST_KEY, objSite.WebApplication, objList.ID);
      }

      objTest2.number1++;
      objTest2.number2++;
      objTest2.number3++;
      objTest2.number4++;
      objTest2.number5++;
      objTest2.number6++;
      objTest2.number7++;
      objTest2.number8++;
      objTest2.number9++;
      objTest2.number10++;
      objTest2.Update();

      //Use the following code to delete info
      //objTest2.Delete();
      //objTest2.Unprovision();

     }
     objTimer2.Stop();

     SPFolder objFolder = objList.RootFolder;
     objTimer3.Start();
     for (int i = 0; i < intRuns; i++)
     {
      int int1 = 1;
      int int2 = 1;
      int int3 = 1;
      int int4 = 1;
      int int5 = 1;
      int int6 = 1;
      int int7 = 1;
      int int8 = 1;
      int int9 = 1;
      int int10 = 1;

      if (objFolder.Properties.Contains("number1"))
      {
       int1 = Convert.ToInt32(objFolder.Properties["number1"]);
       int1++;
      }
      objFolder.Properties["number1"] = int1;

      if (objFolder.Properties.Contains("number2"))
      {
       int2 = Convert.ToInt32(objFolder.Properties["number2"]);
       int2++;
      }
      objFolder.Properties["number1"] = int1;
      if (objFolder.Properties.Contains("number1"))
      {
       int1 = Convert.ToInt32(objFolder.Properties["number1"]);
       int1++;
      }
      objFolder.Properties["number2"] = int2;

      if (objFolder.Properties.Contains("number3"))
      {
       int3 = Convert.ToInt32(objFolder.Properties["number3"]);
       int3++;
      }
      objFolder.Properties["number3"] = int3;

      if (objFolder.Properties.Contains("number4"))
      {
       int4 = Convert.ToInt32(objFolder.Properties["number4"]);
       int4++;
      }
      objFolder.Properties["number4"] = int4;

      if (objFolder.Properties.Contains("number5"))
      {
       int5 = Convert.ToInt32(objFolder.Properties["number5"]);
       int5++;
      }
      objFolder.Properties["number5"] = int5;

      if (objFolder.Properties.Contains("number6"))
      {
       int6 = Convert.ToInt32(objFolder.Properties["number6"]);
       int6++;
      }
      objFolder.Properties["number6"] = int6;

      if (objFolder.Properties.Contains("number7"))
      {
       int7 = Convert.ToInt32(objFolder.Properties["number7"]);
       int7++;
      }
      objFolder.Properties["number7"] = int7;

      if (objFolder.Properties.Contains("number8"))
      {
       int8 = Convert.ToInt32(objFolder.Properties["number8"]);
       int8++;
      }
      objFolder.Properties["number8"] = int8;

      if (objFolder.Properties.Contains("number9"))
      {
       int9 = Convert.ToInt32(objFolder.Properties["number9"]);
       int9++;
      }
      objFolder.Properties["number10"] = int10;

      objFolder.Update();
     }
     objTimer3.Stop();

     Console.WriteLine(objTimer1.ElapsedMilliseconds);
     Console.WriteLine(objTimer2.ElapsedMilliseconds);
     Console.WriteLine(objTimer3.ElapsedMilliseconds);
    }
   }
   catch (Exception err)
   {
    Console.Write(err.Message);
   }
  }

  /// <summary>
  /// To convert a Byte Array of Unicode values (UTF-8 encoded) to a complete String.
  /// </summary>
  /// <param name="characters">Unicode Byte Array to be converted to String</param>
  /// <returns>String converted from Unicode Byte Array</returns>
  //taken from: http://www.dotnetjohn.com/articles.aspx?articleid=173
  private static String UTF8ByteArrayToString(Byte[] characters)
  {
   UTF8Encoding encoding = new UTF8Encoding();
   String constructedString = encoding.GetString(characters);
   return (constructedString);
  }

  /// <summary>
  /// Converts the String to UTF8 Byte array and is used in De serialization
  /// </summary>
  /// <param name="pXmlString"></param>
  /// <returns></returns>
  //taken from: http://www.dotnetjohn.com/articles.aspx?articleid=173
  private static Byte[] StringToUTF8ByteArray(String pXmlString)
  {
   UTF8Encoding encoding = new UTF8Encoding();
   Byte[] byteArray = encoding.GetBytes(pXmlString);
   return byteArray;
  }

  public static Object DeserializeObject(String pXmlizedString)
  {
   XmlSerializer xs = new XmlSerializer(typeof(SerializeClass));
   MemoryStream memoryStream = new MemoryStream(StringToUTF8ByteArray(pXmlizedString));
   XmlTextWriter xmlTextWriter = new XmlTextWriter(memoryStream, Encoding.UTF8);
   return xs.Deserialize(memoryStream);
  }
}
}

In the next section (called "Results") we’ll discuss what we’ve found when executing this test harness.

Results

The results of our test runs were, at least to us, quite surprising. The results are shown in the next table. It contains the number of runs. For instance, we’re updating and saving 10 properties. If we repeat this process 5 times, the number of test runs is 5.

(runs no)\(scenario)	XML serialize	Hierarchical store	Multiple props
2 runs	1689 msec	302 msec	46 msec
2 runs	1695 msec	334 msec	66 msec
10 runs	1887 msec	1398 msec	123 msec
10 runs	2122 msec	983 msec	275 msec
50 runs	2267 msec	4828 msec	602 msec
50 runs	2314 msec	CRASH	-

As you can see, XML serialization isn’t that fast, but it doesn’t become much slower either. It’s pretty stable and in the end, doing 50 test runs, it surpasses the performance of the hierarchical object store. Storing values in multiple properties in the property bag of the root folder turns out to be very fast and it’s stable too. The hierarchical object store is, at first, a lot faster than XML serialization, but never as fast as the multiple properties scenario. This mechanism isn’t quite as stable as the other two, its test performance results tend to fluctuate more. Then, when doing 50 test runs, the hierarchical object store scenario crashes badly and stays crashed, resulting in the following error message:

"An update conflict has occurred, and you must re-try this action. The object PersistStore Name=TestAppSetting Parent=SPWebApplication Name=[name of web application] is being updated by [domain]\[username], in the HObjectStore.vshost process, on machine [server name]. View the tracing log for more information about the conflict."

Based on these results, we’re guessing you’re better of storing site and list configuration info in the property bag instead of in the hierarchical object store as it seems to offer better scalability and more robustness.

Technorati Profile

Working with noise words - update

04 August 2008 08:42:47

Working with noise words - update

Written by: Margriet Bruggeman, Nikander Bruggeman.

In August 2007 we wrote an article about working with noise words. In that article, we included a little script that displays Shared Service Provider GUIDs.

Apparently Mauro Cardarelli has read this article and has improved it. We thought it was a valuable addition, so we’ve included the script here as well: Set objGatherAdmin = WScript.CreateObject("oSearch.GatherMgr.1")
For Each objApplication in objGatherAdmin.GatherApplications
WScript.echo " "
WScript.echo "SSP: " & objApplication.DisplayName & " Application GUID: "
& objApplication.Name
Next

The DisplayName property can be used to map SSPs to application GUIDs.

The Feature Gooney

29 July 2008 13:09:01

The Feature Gooney

Written by: Margriet Bruggeman, Nikander Bruggeman.

Currently we have two hobby community projects: Monk for SharePoint 2007 and the Feature Gooney. Often, it's hard to find the time to add the features to both projects that we'd like and unfortunately both projects have not nearly reached the version 1.0 feature set that we've envisioned for them. Having said that, working on both projects has been a lot of fun and we think both tools in their current state offer value.

Monk has been downloadable for some time now and the current version can be used to produce HTML documentation about your MOSS 2007 installation. The top priorities on our wish list are some bug fixes and the possibility to generate Word (.docx) documentation.

The reason we've decided to start building the Feature Gooney is that we haven't seen a community tool that facilitates working with SharePoint 2007 features that we really really like. The current version of the Feature Gooney is a WPF application that doesn't do much yet, but what is does at this point is COOL (at least, that's what we think)! The Feature Gooney offers a couple of tabs offering different functionality. Currently the following tabs do something:

Configuration
12 Hive Explorer
Miscellaneous

Of these tabs, only the 12 Hive Explorer does something cool. Once you click on it, a treeview is generated containing all folders in the FEATURES folder of the 12 hive. You can expand the folders, select files and the detail pane will show its contents. That's when the cool part starts... If the Feature Gooney recognizes XML elements or attributes within a file, it starts displaying help once you hover over the element or attribute as can be seen in Figure 1.

For now, we're calling this feature IntelliSupport (we’ll change that name if we find out that are legal ramifications attached to using this name ?) which makes reading feature related files a lot easier. Currently, there are lots of XML elements and attributes the Feature Gooney doesn't know about, but you don't have to wait until we update the Feature Gooney (although you could!). You can add additional help yourself by editing the XML file IntelliSupport.xml, located in the same dir as the Feature Gooney itself.

If you want, you can download the Feature Gooney here. Make sure to install it on the server that contains the 12 HIVE and another requirement is that the server needs to have .NET 3.5 installed (because the Feature Gooney is a WPF application). Try it out and let us know what you think!

Building a System.Transactions resource manager for SharePoint

29 July 2008 05:27:05

Building a System.Transactions resource manager for SharePoint

Written by: Margriet Bruggeman, Nikander Bruggeman.

A very important part of MOSS is that it holds semi-structured information. Semi-structured repositories were born out of the need to bring both database worlds and document worlds together, and they hold information that are organized in a loose fashion. SharePoint is an excellent example of an application that handles semi-structured information, which it has been doing since it’s first inception in 2001.

You can recognize semi-structured data via some common traits, such as:

Similar list items are grouped together (in site collections, sites, lists or folders, depending on the similarity).
The name of item containers (such as folders) typically describes its contents.
Metadata is used extensively to describe list items.
List items that are stored in the same item container don’t have to share identical sets of metadata.
The order of list item metadata is unimportant.
Pieces of list item metadata may not be required.

Semi-structured information is the opposite of unstructured information which is data that isn’t structured and isn’t easily readable by a machine, such as audio, video, IM messages, and e-mail messages. Looking through SharePoint glasses, to us, unstructured data is just data that is waiting to get some structure.

Semi-structured information is also the opposite of relational information where several pieces of information have some kind of connection to each other. Or as the British mathematician and logician Augustus De Morgan (not your typical English name, now is it?) has put it beautifully around 150 years ago:

"When two objects, qualities, classes, or attributes, viewed together by the mind, are seen under some connexion, that connexion is called a relation."

Currently, MOSS doesn’t handle relations for list items well, although we hope and expect that this changes in the future, because there are valid reasons why you would need them. For example, you might want to store compound documents, or documents that have relationships to each other (before someone asks, the lookup fields mechanism is too basic to implement document relationships, a topic which we could fill at least a book chapter with, but that’s another story).

As we’ve said, MOSS doesn’t handle relations for list items well, but it does offer the infrastructure that allows you to implement such a system, the list event system being the foremost member of this infrastructure. The list event system allows you to implement scenarios like this:

Item A has a relationship to Item B, so if this piece of metadata for Item A changes this affects Item B as well.
Item A links to Item B, so if Item B moves to another location, the link to Item B needs to be updated as well.
If I remove Item A, Item B needs to be removed as well.

When building a system that supports relations between list items, you’ll soon find out that you need a transaction, quite a well-known concept, especially in the database world. Since MOSS doesn’t support list item transactions, you need to build such a mechanism yourself. We think the best way to do this is to build a System.Transactions resource manager for SharePoint, which is coincidentally the topic of this article. To be very precise, we’ll show how to build a resource manager that’s able to support transactions for list item metadata.

Background info

Before we dive into building System.Transactions resource managers it helps if you know a little bit about transactions and related topics in general. As a quick refresher, we’ll discuss some concepts that are important for a complete understanding of the article. These concepts are:

Transactions
ACID
Two-phase commit protocol (2PC) *
Single-phase commit
MS DTC
Redo log

* For this article this is the most important concept of the topics mentioned above.

Transactions and ACID

A transaction is a unit of work that is performed against some kind of data repository that is treated in a coherent and reliable way independent of other transactions. This means a transaction must have four traits that are often referred to by the acronym ACID. A transaction must be:

Atomic, all tasks within a transaction should be completed or none of them.
Consistent, the data repository should remain in a consistent state before the start of the transaction as well as after the end of the transaction (even if the transaction fails).
Isolated, a transaction should appear isolated from other operations. This means nobody is allowed to see the intermediate state of data during the transaction.
Durable, once the transaction manager notifies the client that the transaction has been successful, the transaction is persisted and cannot be undone (even in the case of a system failure).

Two-phase commit protocol

The 2PC protocol is an algorithm that helps a system coordinate a transaction and is important to us because we will be demonstrating how to build a System.Transactions resource manager for SharePoint that supports this protocol. Tasks (and ultimately resources) within a 2PC transaction are managed by resource managers. All resource managers within a transaction are managed by the transaction coordinator. There are two kinds of resources (and therefore, there are two kinds of resource managers as well):

Durable resources, these are resources whose state is expected to exist after the lifetime of a transaction. In case of a failure, these resources need to be recovered.
Volatile resources, these are resources whose state is not expected to exist after the lifetime of a transaction (such resources only exist in memory). In case of a failure, these resources won’t be recovered.

The 2PC is mostly used within distributed environments and consists of different phases and different steps within those phases. Every transaction that uses the two-phase commit protocol goes through a life cycle of three phases (yes, you might want to re-read that line, because the two-phase commit protocol does have three phases). The following list shows how the 2PC protocol works:

Observation phase (also known as phase Zero), the transaction coordinator observes resource managers that are participating in the transaction.

Resource managers that want to participate in a transaction are enlisted via a transaction coordinator.

Voting phase (also known as phase One), the transaction coordinator tries to gather information from the resource managers participating in the transaction and decides if the transaction is successful or not. Basically, during this phase, resource managers are telling the transaction coordinator if they are able to perform the action we want them to perform.

The transaction coordinator asks all resource managers if they’re ready to commit.
Resource managers perform some kind of action.
Resource managers write to an undo log. The undo log contains a list of all changes that will be reversed (the rollback) if the transaction fails.
Resource managers write to a redo log. The redo log contains a list of all changes made to the underlying data store. Redo logs can contain committed, as well as uncommitted transactions. In this article, we won’t show how to implement a redo log. However, at the end of this section we’ll talk about redo logs some more.
Resource managers reply to the transaction coordinator letting it know whether the transaction was successful. A resource vote can be either one of three things:

Read-only vote, the resource managers agrees to commit its part of the transaction and doesn’t need an outcome notification.
Prepared vote, the resource managers agrees to commit its part of the transaction and needs an outcome notification.
Abort vote, the entire transaction needs to be aborted because the resource managers fails to complete its part of the work.

The transaction coordinator waits a given amount of time until it has responses from all resource managers. Based on this information, the transaction coordinator decides if the transaction has been a success or a failure. If the transaction coordinator doesn’t get all the responses in time, the transaction fails.

Final phase success (the final phase is also known as phase Two). The transaction is completed. During this phase, resources should actually perform the action we want them to perform. At this stage, resource managers should always be able to perform the required action, or else the 2PC protocol gets screwed up (the 2PC protocol isn’t flawless and there are scenarios where it can actually cause problems; at the end of this article we’ve included a link to more information).

If all resource managers completed their tasks, and they’ve let the transaction coordinator know so, the transaction coordinator sends a Commit notification to all resource managers that sent a Prepared vote (that was sent in phase One, thus letting the transaction coordinator know they needed an outcome notification from the transaction coordinator).
Each resource manager that sent a Prepared vote completes any remaining actions required to commit their part of the transaction and releases any locks and objects it holds.
Each resource manager that sent a Prepared vote acknowledges to the transaction coordinator that it has finished its job.
The transaction coordinator completes any remaining actions (if necessary) and releases any locks and objects it holds.

Final phase (error). The transaction has failed and any temporary changes need to be reversed.

If at least one of the resource managers failed their tasks, or if at least one of the resource managers failed to let the transaction coordinator know, the transaction coordinator sends an Abort notification to all resources that sent a Prepared vote.
Each resource manager that sent a Prepared vote rollbacks any changes it performed based on its undo log and releases any locks and objects it holds.
Each resource manager that sent a Prepared vote acknowledges to the transaction coordinator that it has finished its job. If a resource manager loses contact with the transaction coordinator before this happens, the resource manager is said to be "In Doubt". Durable resource managers will try to reconnect to the transaction coordinator later and perform recovery actions. The other way round works too, if a transaction coordinator is still up and running, but loses contact with one or more of the resource managers within the transaction, the coordinator sends an In Doubt notification to the remaining connected resource managers.
The transaction coordinator completes any remaining actions (if necessary) and releases any locks and objects it holds.

As you’ve seen, although the 2PC protocol consists of three phases, it gets its name from the fact that typically every resource manager performs and commits their actions in two steps: it determines if it is able to perform an action, and then asks the transaction coordinator for permission to do it before it permanently commits the changes.

The 2PC describes an algorithm for handling distributed transactions, but it doesn’t provide an implementation of it. If you want that you need to take a look at OLE Transactions (or you can simply call it OleTx), a distributed transaction protocol that is an implementation of the 2PC protocol. If you want to learn more about OleTx, check out http://msdn.microsoft.com/en-us/library/cc229116.aspx ). On Windows platforms, OleTx works in conjunction with the Distributed Transaction Coordinator (DTC, which will be discussed in more detail later in this section).

Single-phase commit

Starting a distributed transaction (for instance, a 2PC transaction handled by the DTC) by default results in a lot of overhead if it turns out you don’t really need a distributed transaction after all. A transaction doesn’t need to be distributed when all resources within a transaction run in the same application domain. Also, such transactions are allowed to contain one or multiple volatile resources (in which case you won’t need an advanced transaction mechanism) or at most a single durable resource (in which case it is okay to let the underlying data repository, such as SQL Server, handle the transaction). In such cases, you can use a protocol that is a bit more lightweight compared to 2PC. The OleTx transaction protocol specifies such a lightweight algorithm, called Single-Phase Commit which works like this:

The transaction won’t be distributed because it contains one or multiple volatile resources or at most a single durable resource. Because of this, a lightweight transaction coordinator is created that coordinates the transaction (for as long as it stays a non-distributed transaction).
The transaction coordinator doesn’t ask resources if they are ready to commit their tasks, instead it asks them to perform a single-phase commit. In doing this, the transaction coordinator effectively delegates the right to decide the transaction outcome to the resources within the transaction. This lack of explicit coordination enhances runtime performance.
Resources accept the delegation of rights and commit (or rollback) their actions. After doing so, they will notify the transaction coordinator.
Alternatively, a resource can decide to reject the delegation of rights by responding with a Prepared vote. In such a case, the transaction coordinator takes care of the transaction outcome decisioning.

By the way, transactions that start out as non-distributed may need to become distributed after all (for instance, if more durable resources are added to the transaction). If this happens, the non-distributed transaction is said to have been promoted to a distributed transaction.

DTC

The 2PC depends upon the availability of a transaction coordinator, which you luckily don’t have to write yourself. Microsoft has been shipping a transaction coordinator of their own that’s able to handle distributed transactions (that span multiple domains, processes or machines) for years now. It is called the Microsoft Distributed Transaction Coordinator service and uses the OleTx distributed transaction protocol. By the way, since .NET 2.0 you have other (newer) choices when it comes to choosing transaction coordinators as well, but we’ll discuss those in section "System.Transactions background".

You can program against the DTC directly, or use it indirectly, for example, via Enterprise Services. The following procedure explains how to monitor the DTC:

Start > Programs > Administrative Tools > Component Services. This opens the Component Services MMC snap-in.
Open the Component Services node.
Open the Computers node.
Open the My Computer node.
Open the Distributed Transaction Coordinator node.
Click on either the Transaction List or Transaction Statistics node if you want to find out what the DTC is doing.

Redo log

We promised that we’d talk more about the redo log at the end of this section. Well, here’s the end of this section, so let’s talk about it. Essentially, a redo log records all changes to a resource to prevent data loss. Durable resources such as a database typically write both to a local redo log and a standby redo log on one or more other (standby) databases. If something really bad happens, the redo log is used for recovering data.

A row in a redo log is called a redo entry that contains so-called change vectors that describe what change has been made to a resource. As a single transaction may instigate multiple changes in a resource, a transaction may be described in multiple redo entries. Redo entries contain both committed and uncommitted transactions. Typical change vectors that are included in a redo entry are:

Indicators that specify when a transaction started.
A unique transaction identifier.
The name of the data object within a resource that was changed (such as the name of a database table).
An image of the data that existed before the change.
An image of the data that existed after the transaction made its changes.
Commit-indicators that indicate whether the transaction has been successful.

A client will only be notified by a resource that the transaction has been completed after the system has successfully updated the redo log file. If the resource crashes, the recovery process tries to apply all (committed and uncommitted) transactions to its data, using the information it finds in the redo log. It must redo all transactions that were committed, and undo all transactions that were uncommitted (by applying the before and after data images, so the transactions from the past are not actually replayed). Of course, redo logs are only useful in scenario’s where you’re working with durable resources, otherwise, you’ll never need to recover data so you won’t need a redo log.

What Microsoft Office SharePoint Server 2007 can’t do for you

MOSS doesn’t support transactions when working with list items. In our opinion, this is something that definitely needs to be added to future versions. This doesn’t mean that MOSS doesn’t support any kind of transactions at all you could use. For example, there are:

Transactions at the SQL Server database level (which should be regarded as off limits, but underwater they happen still the same).
Transactions in workflows (built on the Windows Workflow Foundation framework).
CAML commands wrapped in batches and issued via SharePoint RPC calls.

At the end of this article, we will have created a mechanism that allows you to manipulate metadata of one or more SharePoint list items within one or more transactions. To show a preview of what we’re trying to accomplish and what you can’t achieve with out of the box MOSS functionality, take a look at the following attempt to create a transaction that consist of several actions that manipulate SharePoint list item meta data: using (SPSite site = new SPSite("[URL site collection]"))
{
using (SPWeb web = site.OpenWeb("[URL site]"))
{
  web.AllowUnsafeUpdates = true;
  using (TransactionScope ts = new TransactionScope())
  {
   SPFile objFile1 = web.GetFile("[URL file]");
   SPFolder objFolder1 = web.GetFolder("[URL folder]");

   objFile1.CheckOut();
   objFile1.Properties[strKey] = strValue
   objFile1.Update();
   objFile1.CheckIn(strComment);

   objFolder1.Properties[strKey] = strValue;
   objFolder1.Update();

   ts.Complete();
  }
}
}

Similar code would work great when working with databases such as SQL Server or Oracle, but since the SharePoint object model does not enlist in any transaction coordinator, nothing happens with SharePoint list items when a transaction is rolled back, making it quite useless to create a transaction scope.

System.Transactions background

In this section, we will discuss the basics of what you need to know if you want to build a System.Transactions resource manager in .NET. First of all, if you want to implement some kind of transactional system without resorting to products like Enterprise Services and WCF, it is good to know that since .NET 2.0 two new transaction coordinators have been created that you can use:

The Lightweight Transaction Manager (LTM) which only handles transactions that contain resources that are located within the same application domain.
The OleTx Transaction Manager which can handle transactions that span multiple application domains (including cross-machine calls). Under the covers, this OleTx Transaction Manager handles distributed transactions by leveraging COM+ DTC technology by dynamically configuring a temporary Enterprise Service through Services Without Components (SWC, a COM+ 1.5 feature). This means you can use the DTC management console (discussed in section "DTC") to monitor transactions that are handled by the OleTx Transaction Manager.

Functionality in the System.Transactions namespace takes care of communicating with these transaction coordinator, so you don’t need to interact with them yourself. Earlier, when we talked about Single Phase commit, we discussed that a transaction can get promoted. This also holds true for these two new transaction coordinators: a non-distributed transaction will be handled by the LTM, until the transaction spans multiple durable resources. At that point, the transaction gets promoted and handled by the OleTx Transaction Manager instead. Additionally, a transaction also gets promoted when a transaction object is serialized across an application domain boundary.

If you want either the LTM or OleTx Transaction Manager to handle transactions for you, you need to create a System.Transactions resource manager. The LTM and OleTx Transaction Manager transaction coordinators know how to communicate with System.Transactions resource managers, and if you’re creating a custom System.Transactions resource manager you need to make sure that it is able to communicate back to these transaction coordinators and make sure it is able to handle some kind of resource.

If you want to implement a System.Transactions resource manager that supports the 2PC algorithm you need to implement the IEnlistmentNotification interface within the System.Transactions namespace (in the System.Transactions.dll) which makes it quite easy to implement the 2PC algorithm. If you want to do this you need to make sure of the following:

You need to enlist the resource manager in the transaction via the transaction coordinator. This is a step that is a part of the observation phase (phase Zero) of the 2PC and is not described in the IEnlistmentNotification interface, so you need to take care of that yourself by calling one of the available Enlist() methods of the current transaction (we’ll show how to do that later). Here you have three choices:

Call the EnlistDurable() method if you want to enlist a durable resource manager.
Call the EnlistVolatile() method if you want to enlist a volatile resource manager.
Call the EnlistPromotableSinglePhase() method if you want to enlist a new transaction coordinator that supports Single Phase commit and transaction promotion. This new transaction coordinator handles the transaction for the resource manager until it decides the transaction should be promoted to another (distributed) transaction coordinator. In System.Transactions terminology, this is called Promotable Single Phase Enlistment or PSPE.

The transaction coordinator calls the Prepare() method of the IEnlistmentNotification interface during the Voting phase (phase One) to ask if all System.Transactions resource managers in the transaction are willing and able to perform the action they’re responsible for. So, your System.Transactions resource manager needs to implement the Prepare() method. In this method you should perform some kind of action that is part of the transaction, write to an undo log that allows you to revert changes, write to a redo log (only if you’re managing a durable resource) that contains a list of all changes and send a reply to the transaction coordinator (either a Read-only, Prepared or Abort vote).
The transaction coordinator calls the Commit() method of the IEnlistmentNotification interface of every resource manager that sent a Prepared vote (which lets the transaction coordinator know the System.Transactions resource manager wants an outcome notification) if the transaction coordinator decides the transaction should be committed. So, your System.Transactions resource manager needs to implement this Commit() method. In this method you should complete any remaining actions to commit your part of the transaction, release any locks and objects the resource manager is holding and let the transaction coordinator know you’re finished.
The transaction coordinator calls the Rollback() method of the IEnlistmentNotification interface of every resource manager that sent a Prepared vote if the transaction coordinator decides the transaction should be aborted. So, your System.Transactions resource manager needs to implement this Rollback() method. In this method you should rollback any changes based on the undo log of your resource manager (which you need to create and update yourself). After that, you should release any locks and objects the resource manager is holding and let the transaction coordinator know you’re finished.
The transaction coordinator calls the InDoubt() method of the IEnlistmentNotification interface of every resource manager that sent a Prepared vote if the transaction coordinator loses contact with one or more resource managers. It’s up to you to decide what the System.Transactions resource manager does in this situation.

Enlisting a System.Transactions resource manager

We’ve already discussed that if you want to enlist a System.Transactions resource manager with the current transaction, there are three options available to you. Here, we’ll preview how to enlist a volatile System.Transactions resource manager. Later on, we’ll see extensive examples of all options. The following code listing checks if there is a current transaction, and if so, enlists a volatile resource manager with it: Transaction tran = Transaction.Current;
if (tran != null)
{
tran.EnlistVolatile(this, EnlistmentOptions.None);
}

Implementing Prepare()

The transaction coordinator passes a PreparingEnlistment object (called preparingEnlistment) to this method that allows you to communicate with the transaction coordinator. You should call its Prepared() method if the System.Transactions resource manager is able to perform its part of the transaction, like so: preparingEnlistment.Prepared(); If the System.Transactions resource manager is not able to perform its work, it should force a rollback, like so: preparingEnlistment.ForceRollback(); You can also cast a read-only vote, indicating that the System.Transactions resource manager is committing its part of the transaction but doesn’t need an outcome notification. After casting the read-only vote, the System.Transactions resource manager won’t receive further notifications from the transaction manager. You can cast the read-only vote like so: preparingEnlistment.Done()

Committing

The transaction coordinator passes an Enlistment object (called enlistment) to the Commit() method that allows you to communicate with the TM. At this stage, the System.Transactions resource manager should always be able to perform the required action and indicate it has done so, or else the 2PC protocol gets screwed up (see http://msdn.microsoft.com/en-us/library/system.transactions.enlistment.done.aspx for more information). Because of this, it is very important to always call the following method: enlistment.Done();

Rollback

The transaction coordinator passes an Enlistment object (called enlistment) to the Rollback() method that allows you to communicate with the transaction coordinator. Here, you should implement your custom rollback mechanism and let the transaction coordinator know that you’re finished by calling: enlistment.Done();

InDoubt

The transaction coordinator passes an Enlistment object (called enlistment) to the InDoubt() method that allows you to communicate with the transaction coordinator. Here, you should add your own implementation for dealing with the InDoubt state and let the transaction coordinator know that you’re finished by calling: enlistment.Done();

IPromotableSinglePhaseNotification interface

If you want to implement a custom non-distributed transaction coordinator that supports the Single Phase Commit algorithm and transaction promotion (so that the custom transaction coordinator is able to escalate the transaction to a distributed transaction coordinator, also known as PSPE which was discussed in the beginning of section "System.Transactions background") you need to implement the IPromotableSinglePhaseNotification interface, which contains the following methods:

Initialize, notifies transaction participants that enlistment has been completed.
Rollback, notifies transaction participants that the transaction is aborted.
SinglePhaseCommit, notifies transaction participants that the transaction is committed.

The IPromotableSinglePhaseNotification interface inherits from the ITransactionPromoter interface, which defines a single method called Promote(). Once this method is called the custom transaction coordinator needs to produce a propagation token (in the form of a byte array) which will be used by the next transaction coordinator to obtain a clone of the current transaction. We’ll show how to do this later.

If you implement the IPromotableSinglePhaseNotification interface you’ve basically created a custom transaction coordinator that allows a System.Transactions resource manager to say to the custom transaction coordinator: could you please take over the burden of taking care of this transaction for me?

Later on in this article, we’ll take this quite literally as we’ll create a custom transaction coordinator that doesn’t give it’s enlisted resource managers a whole lot of influence anymore in the proceeding of a transaction and makes decisions for all transaction participants that have enlisted on a transaction handled by the custom transaction coordinator.

The ISinglePhaseNotification interface

If you want to create a System.Transactions resource manager that supports Single Phase Commit, you’ll need to implement the ISinglePhaseNotification interface. This interface inherits from the IEnlistmentNotification interface, which we’ve already discussed. It makes sense that this is so, because resource managers supporting Single Phase commit and transaction promotion could always end up being a part of a 2PC transaction. For instance, this could happen because several durable System.Transactions resource managers are part of the transaction.

The ISinglePhaseNotification interface defines a new method called SinglePhaseCommit() which is called if the transaction coordinator chooses to use Single Phase commit, thereby delegating the right to decide the transaction outcome to the System.Transactions resource manager. The transaction coordinator passes an SinglePhaseEnlistment object (called singlePhaseEnlistment) to the System.Transactions resource manager that allows you to communicate with the transaction coordinator. The System.Transactions resource manager is responsible for letting the transaction coordinator know which decision it has made, and it has a couple of choices: Abort, Commit or Done.

If you want to let the transaction coordinator know that you’ve decided to abort the operation, you need to call: singlePhaseEnlistment.Aborted();

If you want to let the transaction coordinator know that you’ve decided to commit the operation, you need to call: singlePhaseEnlistment.Committed();

If you want to let the transaction coordinator know that you’ve decided to commit the operation and don’t want to receive any further notifications from the transaction coordinator , you need to call: singlePhaseEnlistment.Done();

And via the last option you can let the transaction coordinator know that the System.Transactions resource manager thinks the transaction status is in doubt (hopefully it doesn’t think so because it can’t reach the transaction coordinator anymore, otherwise this call is going to get a bit funky), by calling: singlePhaseEnlistment.InDoubt();

Creating a System.Transactions resource manager for SharePoint

Now that we’ve covered the theory behind all this stuff extensively, let’s write some code. To test the code in this article, all you need to do is create a console application that has references to the Microsoft.SharePoint.dll and System.Transactions dll. We’ve created our test app using VS.NET 2008, so if you’re still using VS.NET 2005, you might have some problems with the reference to Linq (you can just remove it, since we won’t be using Linq anyway).

At first, we will be building two volatile resource managers for MOSS. One of them is called VolatileFileCommand and manages metadata for files in SharePoint lists. The other one is called VolatileFolderCommand and manages metadata for folders in SharePoint lists. Since both of them share common traits, we’ve decided to create a base class for both of them called VolatileMossResourceManager. So, let’s start the discussion with this base class.

Implementing volatile resource managers

First of all, since we’re creating a System.Transactions resource manager you need to import the System.Transactions namespace. As we’re creating a resource manager that supports the 2PC you’ll also need to implement the IEnlistmentNotification interface. When implementing this interface, you need to make sure that you enlist this resource manager in the current transaction. Since we’ve decided to create a volatile resource manager, you need to call the EnlistVolatile() method of the current transaction. To support all this we’ve created a method called EnlistTransaction() that is called by our resource manager whenever it sees fit to do so (but it has to be during phase 0). It looks like this: public void EnlistTransaction() { if (IsTransactionEnlisted) return;

Transaction tran = Transaction.Current;
if (tran != null)
{
tran.EnlistVolatile(this, EnlistmentOptions.None);
}

IsTransactionEnlisted = true;
}

During phase 1, the transaction coordinator calls the Prepare() method of every System.Transactions resource manager. As we’ll see later, at this time our file and folder System.Transactions resource managers will already have performed the actions they needed to perform as part of the transaction, they will also have updated the undo log and since it’s not a durable resource we don’t need a redo log. So all that’s left to do is notify the transaction coordinator that we’re willing to go ahead with the transaction by sending a Prepared vote. The implementation of our Prepare() method looks like this: public virtual void Prepare(PreparingEnlistment preparingEnlistment)
{
preparingEnlistment.Prepared();
}

We’ll leave the implementations of the Commit() and Rollback() methods to the file and folder child classes and won’t implement the InDoubt() method as this is overkill for volatile System.Transactions resource managers and way too much work to implement meaningfully for durable System.Transactions resource managers (in all examples we ever saw about the System.Transactions namespace nobody ever implemented this method, which is no coincidence).

Finally, we’ll use a dictionary object and use it as an Undo log and create a custom method called SaveOrgValue() that saves values in this Undo log. We’ll also add two flags. The first is called IsTransactionEnlisted, a flag that keeps track if transaction enlistment has already taken place. The other one is called MetadataIsDirty, a flag that monitors if actual list item metadata has been changed.

The next code listing shows the complete implementation of our resource manager base class called VolatileMossResourceManager. using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;

namespace TxTest.MossTx.Volatile
{
public abstract class VolatileMossResourceManager : IEnlistmentNotification
{
  public void EnlistTransaction()
  {
   if (IsTransactionEnlisted) return;

   Transaction tran = Transaction.Current;
   if (tran != null)
   {
    tran.EnlistVolatile(this, EnlistmentOptions.None);
   }

   IsTransactionEnlisted = true;
  }

#region IEnlistmentNotification Members
  public abstract void Commit(Enlistment enlistment);

  public void InDoubt(Enlistment enlistment)
  {
   // Do nothing.
  }

  public virtual void Prepare(PreparingEnlistment preparingEnlistment)
  {
   preparingEnlistment.Prepared();
  }

  public abstract void Rollback(Enlistment enlistment);
#endregion

  protected void SaveOrgValue(string strKey, string strOldValue)
  {
   if (!UndoLog.ContainsKey(strKey))
   {
    UndoLog.Add(strKey, strOldValue);
   }
  }

#region props
  private bool _blnIsTransactionEnlisted;
  public bool IsTransactionEnlisted
  {
   get { return _blnIsTransactionEnlisted; }
   set { _blnIsTransactionEnlisted = value; }
  }

  private bool _blnMetadataIsDirty;
  public bool MetadataIsDirty
  {
   get { return _blnMetadataIsDirty; }
   set { _blnMetadataIsDirty = value; }
  }

  private Dictionary _objUndoLog = new Dictionary();
  public Dictionary UndoLog
  {
   get
   {
    return _objUndoLog;
   }
   set
   {
    _objUndoLog = value;
   }
  }
#endregion

}
}

Implementing a volatile file resource manager

The first concrete resource manager that we’re creating is the volatile File System.Transactions resource manager. It inherits from the base class VolatileMossResourceManager and also implements the IEnlistmentNotification interface. We’ll also make sure it has a reference to an SPFile object so that our resource manager is able to interact with MOSS.

It has a custom method called SetValue() which is called by a client whenever it needs to update file metadata. This method does several things:

It checks if the resource manager is already enlisted in the transaction. If this is not so, it enlists the System.Transactions resource manager.
It tries to lock the current file so that the System.Transactions resource manager has exclusive access to it. If this fails, the System.Transactions resource manager will indicate that the transaction should abort.
It saves metadata changes to the Undo log.
It updates the file metadata.

The implementation of the SetValue() method looks like this: public void SetValue(string strKey, string strValue)
{
EnlistTransaction();
LockFile();
SaveOrgValue(strKey, File.Properties[strKey].ToString());
File.Properties[strKey] = strValue;
}

We won’t discuss the methods it’s calling as we think they are self-explaining. We’ll show them later on and we guess you’ll have no trouble figuring out what they’re doing. There are two points of interest left that we need to discuss: the implementations of the Commit() and Rollback() methods.

In the Commit() method we’ll check-in the file we’re working with. This should be no problem, since we know we’ve successfully acquired a file lock earlier on. Then, we should let the transaction coordinator know we’ve finished. The Commit() method looks like this: public override void Commit(Enlistment enlistment)
{
CheckIn("System.Transaction manager commits transaction");
enlistment.Done();
}

The Rollback method is also pretty simple. If we’ve been successful in acquiring a file lock, all we need to do is undo the check out to rollback our changes. If we weren’t successful when acquiring a lock, we haven’t done any changes at all, so we’re also done with the rollback. Finally, we should let the transaction coordinator know we’ve finished. The Rollback() method looks like this: public override void Rollback(Enlistment enlistment)
{
if (LockedFile)
{
File.UndoCheckOut();
}
enlistment.Done();
}

The next code listing shows the complete implementation of the volatile file resource manager for MOSS: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;

namespace TxTest.MossTx.Volatile
{
public class VolatileFileCommand : VolatileMossResourceManager, IEnlistmentNotification
{
#region ctor
  public VolatileFileCommand(SPFile objFile)
  {
   File = objFile;
  }
#endregion

public void SetValue(string strKey, string strValue)
{
  EnlistTransaction();
  LockFile();
  SaveOrgValue(strKey, File.Properties[strKey].ToString());
  File.Properties[strKey] = strValue;
}

private void LockFile()
{
  if (!MetadataIsDirty)
  {
   if (File.CheckOutStatus != SPFile.SPCheckOutStatus.None) throw new Exception("Can't lock file");
   File.CheckOut();
   MetadataIsDirty = true;
   LockedFile = true;
  }
}

public void CheckOut()
{
  File.CheckOut();
}

public void CheckIn(string strComment)
{
  File.CheckIn(strComment);
}

public void Update()
{
  File.Update();
}

public override void Commit(Enlistment enlistment)
{
  CheckIn("System.Transaction manager commits transaction");
  enlistment.Done();
}

public override void Rollback(Enlistment enlistment)
{
  if (LockedFile)
  {
   File.UndoCheckOut();
  }
  enlistment.Done();
}

#region props
private SPFile _objFile;
public SPFile File
{
  get
  {
   return _objFile;
  }
  set
  {
   _objFile = value;
  }
}

private bool _blnLockedFile;
public bool LockedFile
{
  get { return _blnLockedFile; }
  set { _blnLockedFile = value; }
  }
#endregion
}
}

As you may remember, we’ve put all our code in a console application. In the next code listing we’ll use it to obtain a valid reference to a file located in a SharePoint list. Then, we’ll start a new transaction, update a piece of metadata and commit the transaction. Please note that you should always explicitly call the Complete() method of the current transaction, otherwise the transaction will abort. The complete code listing looks like this: static void Main(string[] args)
{
VolatileFileCommand objCommand1;

try
{
  using (SPSite site = new SPSite("http://jupiter"))
  {
   using (SPWeb web = site.OpenWeb("/SiteA/SiteB"))
   {
    SPFile objFile1 = web.GetFile("http://myserver/SiteA/SiteB/ListC/DocA.doc");

    using (TransactionScope ts = new TransactionScope())
    {
     objCommand1 = new VolatileFileCommand(objFile1);
     objCommand1.SetValue("Dossiernummer", "value " + DateTime.Now);
     objCommand1.Update();

     ts.Complete();
    }
   }
  }
  Console.Write("Completed");
}
catch (Exception err)
{
  Console.Write(err.Message);
}
Console.ReadLine();
}

This is about the simplest example of using the file System.Transactions resource manager that we can think of. If the transaction is successful, its life cycle looks like this:

The constructor of the VolatileFileCommand object stores a reference to a valid SPFile object.
Before metadata is changed, the System.Transactions resource manager is enlisted in the current transaction.
The file stored in the SharePoint list is checked out.
An entry is added to the Undo log.
A piece of metadata for the file is changed.
The file is updated (and persisted in the SharePoint content database).
The transaction is completed.
The transaction coordinator calls the Prepare() method of the System.Transactions resource manager, that in turn notifies the transaction coordinator that it’s ready to finish the transaction.
The transaction coordinator calls the Commit() method of the System.Transactions resource manager, that performs left-over jobs. This method checks in the file and notifies the transaction coordinator that it has finished.

If the transaction is aborted, the life cycle looks a bit different. We’ll take a look at an example scenario. Let’s suppose the file is already checked out by somebody else. In this case, the life cycle looks like this:

The constructor of the VolatileFileCommand object stores a reference to a valid SPFile object.
Before metadata is changed, the System.Transactions resource manager is enlisted in the current transaction.
The System.Transactions resource manager tries to lock the file, but its attempt to check out the file fails, and an exception is thrown.
The transaction coordinator calls the Rollback() method, which checks if the file was checked out. Since it was not, we know no changes have been made so we notify the transaction coordinator that we've concluded our part of the transaction rollback.

Implementing a volatile folder resource manager

Next, we’ll discuss the folder System.Transactions resource manager. We’ll take a closer look at the Commit() and Rollback() methods before showing you the entire code listing. Since it’s not possible to check out folders and we’ve chosen to allow folder updates at an early stage (the only way to make sure that we’re indeed able to commit our part of the transaction), there’s not much work left to be done in the Commit() method, except for letting the transaction coordinator know we’re good to go. The next code listing shows our implementation of the Commit() method: public override void Commit(Enlistment enlistment)
{
enlistment.Done();
}

The Rollback() method has become a bit more cumbersome, since we’re not able to discard a check out anymore. In this implementation, we’re using the Undo log to restore the original values. This is shown in the next code listing: public override void Rollback(Enlistment enlistment)
{
foreach (string strKey in UndoLog.Keys)
{
Folder.Properties[strKey] = UndoLog[strKey];
}

Folder.Update();

enlistment.Done();
}

The entire implementation of the folder System.Transactions resource manager is shown in the next code listing: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;

namespace TxTest.MossTx.Volatile
{
public class VolatileFolderCommand : VolatileMossResourceManager, IEnlistmentNotification
{
#region ctor
  public VolatileFolderCommand(SPFolder objFolder)
  {
   Folder = objFolder;
  }
#endregion

  public void SetValue(string strKey, string strValue)
  {
   EnlistTransaction();

   if (Folder.Properties.Contains(strKey))
   {
    SaveOrgValue(strKey, Folder.Properties[strKey].ToString());
   }
   else
   {
    SaveOrgValue(strKey, String.Empty);
   }

   Folder.Properties[strKey] = strValue;
  }

  public void Update()
  {
   Folder.Update();
  }

  public override void Commit(Enlistment enlistment)
  {
   enlistment.Done();
  }

  public override void Rollback(Enlistment enlistment)
  {
   foreach (string strKey in UndoLog.Keys)
   {
    Folder.Properties[strKey] = UndoLog[strKey];
   }

   Folder.Update();

   enlistment.Done();
  }

#region props
  private SPFolder _objFolder;
  public SPFolder Folder
  {
   get
   {
    return _objFolder;
   }
   set
   {
    _objFolder = value;
   }
  }
#endregion

}
}

A client can leverage the folder System.Transactions resource manager like so: VolatileFolderCommand objCommand4;
SPFolder objFolder1 = web.GetFolder("http://myserver/site/doclib/folderA");

using (TransactionScope ts = new TransactionScope())
{
objCommand4 = new VolatileFolderCommand(objFolder1);
objCommand4.SetValue("APieceOfMetadata", "value " + DateTime.Now);
objCommand4.Update();

ts.Complete();
}

This is as simple as working with the folder System.Transactions resource manager will get. Since this is all very similar to the file System.Transactions resource manager scenario, we won’t discuss the life cycle of this code.

Multiple metadata updates and multiple commands

In the next example, we’ll show how to update multiple pieces of metadata and how to work with multiple commands within a single transaction. using (TransactionScope ts = new TransactionScope())
{
objCommand1 = new VolatileFileCommand(objFile1);
objCommand1.SetValue("PropA", "value " + DateTime.Now);
objCommand1.SetValue("PropA", "another value" + DateTime.Now);
objCommand1.SetValue("PropB", "value of related docs" + DateTime.Now);
objCommand1.Update();

objCommand2 = new VolatileFileCommand(objFile2);
objCommand2.SetValue("PropA", "command 2 value " + DateTime.Now);
objCommand2.Update();

ts.Complete();
}

If this transaction is successful, the transaction coordinator first calls the Prepare() method of command 1, followed by a call of the Prepare() method of command 2. After that, it calls the Commit() method of the first command, followed by a call to the Commit() method of the second command.

Nested transactions

Every transaction has a scope to which it applies, and it’s also possible to nest transactions. The way transactions behave when nested is determined by setting their transaction scopes. There are three possible transaction scopes:

Required, this is the default transaction scope option. If a transaction already exists, the TransactionScope object (used extensively in the previous examples) joins that transaction. Otherwise, it creates a new transaction.
RequiresNew, this transaction scope option always starts a new transaction.
Suppress, the TransactionScope object will never be a part of a transaction. This one should be used when you’re performing actions that are nice when they succeed, but you don’t want to abort the entire transaction if they fail.

Using nested transactions affect the way transactions behave. We’ll explore this in the next example: using (TransactionScope ts = new TransactionScope(TransactionScopeOption.Required))
{
using (TransactionScope ts1 = new
TransactionScope(TransactionScopeOption.Required))
{
  // If ts fails, ts1 will fail.
  // If ts1 fails, ts will fail.
  // If ts1 succeeds, ts could succeed.

  // If ts1 fails, the command is rollbacked immediately.
  // If ts1 succeeds, it is committed after ts has completed.

  // Let file or folder System.Transactions resource managers do work...
  ts1.Complete();
}

using (TransactionScope ts2 = new TransactionScope(TransactionScopeOption.RequiresNew))
{
  // If ts fails, ts2 will fail.
  // If ts2 fails or succeeds, it won’t affect ts.

  // If ts2 fails, the ts2 transaction is aborted immediately.
  // If ts2 succeeds, it is committed immediately.

  // Let file or folder System.Transactions resource managers do work...
  ts2.Complete();
}

using (TransactionScope ts3 = new TransactionScope(TransactionScopeOption.Suppress))
{
  // Doesn't participate in any transaction,
  // therefore, it doesn't affect any other transactions,
  // and it doesn't matter if you complete ts3 or not since no parts of the 2PC will
  // be called by the TC..
}

ts.Complete();
}

Implementing a read-only volatile file command

Please note that the current implementations expect the System.Transactions resource manager to participate fully in the 2PC. If you would create a new class called ReadOnlyVolatileFileCommand that inherits from VolatileFileCommand and implements the Prepare() method differently, like so: public override void Prepare(PreparingEnlistment preparingEnlistment)
{
preparingEnlistment.Done();
}

Thereby indicating that the System.Transactions resource manager will commit its changes but no longer participates in the 2PC, its Commit() and Rollback() methods shall never be called by the transaction coordinator. In this particular implementation, that would cause problems as we’re using those methods to release the file lock we’ve placed on the file in a SharePoint list. The complete code for such a class is shown in the following code listing: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;

namespace TxTest.MossTx.Volatile
{
public class ReadOnlyVolatileFileCommand : VolatileFileCommand, IEnlistmentNotification
{
#region ctor
  public ReadOnlyVolatileFileCommand(SPFile objFile) : base(objFile)
  {
  }
#endregion

  public override void Prepare(PreparingEnlistment preparingEnlistment)
  {
   preparingEnlistment.Done();
  }
}
}

The following code listing demonstrates the use of our new class. If you run it, you’ll find out that the Commit() and Rollback() methods of the ReadOnlyVolatileFileCommand class aren’t called by the transaction coordinator, causing File 1 to remain in a checked out state. using (TransactionScope ts = new TransactionScope())
{
objCommand5 = new ReadOnlyVolatileFileCommand(objFile1);
objCommand5.SetValue("Dossiernummer", "value " + DateTime.Now);
objCommand5.SetValue("Dossiernummer", "another value" + DateTime.Now);
objCommand5.SetValue("Gerelateerde documenten", "value of related docs" + DateTime.Now);
objCommand5.Update();

objCommand2 = new VolatileFileCommand(objFile2);
objCommand2.SetValue("Dossiernummer", "command 2 value " + DateTime.Now);
objCommand2.Update();

ts.Complete();
}

Creating a durable resource manager

You can also create durable resource managers. If you want to do so, you need to change the enlistment process a little bit, and make sure you call the EnlistDurable() method of the current transaction object. This method expects a GUID that uniquely defines the current transaction and the current System.Transactions resource manager, so you’ll need to pass such a GUID. The following code shows an example that demonstrates how to enlist a durable resource manager: Transaction tran = Transaction.Current;
if (tran != null)
{
tran.EnlistDurable(MyGuid, this, EnlistmentOptions.None);
}

A durable resource manager should be able to recover from failure. That’s why you need to pass along a GUID (also known as the resource manager identifier), so that the durable resource manager can use it in case of an emergency, such as a resource manager failure or a reboot, to retrieve recovery information. Because of this you’ll need to persist and keep track of these GUIDs.

In our implementation of a durable resource manager we won’t add recovery support (although we will discuss how to implement it in section "Recovery support"). Because of this, we’ll just generate a new GUID every time we’re enlisting a new durable resource manager and we won’t bother to keep these GUIDs safely stored away somewhere.

Please note that you also need to choose which enlistment options you want to use. By default, you should set this option to None. Only if you need to the System.Transactions resource manager to perform additional work during the Prepare phase (Phase 0) you should change this by setting EnlistmentOptions.EnlistDuringPrepareRequired. By setting this parameter, the System.Transactions resource manager indicates that it wants to receive a Prepare notification while new enlistments are still allowed for the transaction.

Apparently you can do some advanced stuff using this option, for instance, you could create a caching resource manager (which is mentioned briefly on http://blogs.msdn.com/florinlazar/archive/2006/01/29/518956.aspx ). The caching resource manager could use the Prepare notification to decide that it needs to transfer its cached contents to a durable resource, such as a database. By doing this, the durable resource enlists on the transaction and also becomes a part of it.

If you don’t set enlistment options (EnlistmentOptions.None) you will receive a Prepare notification once no new enlistments will be accepted by the transaction coordinator. The aforementioned caching resource manager would try to persist its cache, the durable resource would try to enlist on the transaction and this would result in an exception since no enlistments are allowed anymore at this stage.

All in all it’s pretty useless to deviate from the default EnlistmentOptions.None mode, unless you have some advanced motives for being a deviant deviator.

In order to support durable enlistment, we’ve created a new base class called DurableMossResourceManager that has a property called TransactionGuid. The durable resource manager creates a GUID that uniquely identifies itself and the current transaction, and passes it to the transaction coordinator during enlistment. The complete implementation of the DurableMossResourceManager is almost identical to the VolatileMossResourceManager and looks like this: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;

namespace TxTest.MossTx.Durable
{
public abstract class DurableMossResourceManager : IEnlistmentNotification
{
  public void EnlistTransaction()
  {
   if (IsTransactionEnlisted) return;

   Transaction tran = Transaction.Current;
   if (tran != null)
   {
    tran.EnlistDurable(TransactionGuid, this, EnlistmentOptions.None);
   }

   IsTransactionEnlisted = true;
  }

#region IEnlistmentNotification Members
  public void Commit(Enlistment enlistment)
  {
   enlistment.Done();
  }

  public void InDoubt(Enlistment enlistment)
  {
   // Do nothing.
  }

  public virtual void Prepare(PreparingEnlistment preparingEnlistment)
  {
   preparingEnlistment.Prepared();
  }

  public abstract void Rollback(Enlistment enlistment);
#endregion

  protected void SaveOrgValue(string strKey, string strOldValue)
  {
   if (!UndoLog.ContainsKey(strKey))
   {
    UndoLog.Add(strKey, strOldValue);
   }
  }

#region props
  private bool _blnIsTransactionEnlisted;
  public bool IsTransactionEnlisted
  {
   get { return _blnIsTransactionEnlisted; }
   set { _blnIsTransactionEnlisted = value; }
  }

  private bool _blnMetadataIsDirty;
  public bool MetadataIsDirty
  {
   get { return _blnMetadataIsDirty; }
   set { _blnMetadataIsDirty = value; }
  }

  private Dictionary _objUndoLog = new Dictionary();
  public Dictionary UndoLog
  {
   get { return _objUndoLog; }
   set { _objUndoLog = value; }
  }

  private Guid _objTransactionGuid = Guid.NewGuid();
  private Guid TransactionGuid
  {
   get { return _objTransactionGuid; }
   set { _objTransactionGuid = value; }
  }
#endregion
}
}

If you enlist a resource as being durable within a transaction, the transaction will immediately be promoted to a distributed transaction and handled by the DTC. Because it is a durable transaction, we’ll also implement the file resource manager a bit different, so that it only locks files for a very short time (as opposed to locking the file for the entire transaction, as we did when we created our volatile file resource manager). The next code listing, which resembles the previous volatile implementation a lot, shows the code for our durable file System.Transactions resource manager. using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;

namespace TxTest.MossTx.Durable
{
public class DurableFileCommand : DurableMossResourceManager, IEnlistmentNotification
{
#region ctor
  public DurableFileCommand(SPFile objFile)
  {
   File = objFile;
  }
#endregion

  public void SetValue(string strKey, string strValue)
  {
   EnlistTransaction();
   SaveOrgValue(strKey, File.Properties[strKey].ToString());
   File.Properties[strKey] = strValue;
  }

  public void CheckOut()
  {
   File.CheckOut();
  }

  public void CheckIn(string strComment)
  {
   File.CheckIn(strComment);
  }

  public void Update()
  {
   File.Update();
  }

  public override void Rollback(Enlistment enlistment)
  {
   if (File.CheckOutStatus == SPFile.SPCheckOutStatus.None)
   {
    File.CheckOut();
   }

   foreach (string strKey in UndoLog.Keys)
   {
    File.Properties[strKey] = UndoLog[strKey];
   }

   File.Update();
   File.CheckIn("rollback because of a failed transaction");

   enlistment.Done();
  }

#region props
  private SPFile _objFile;
  public SPFile File
  {
   get { return _objFile; }
   set { _objFile = value; }
  }
#endregion
}
}

Please note that a Durable folder System.Transactions resource manager is basically identical to its volatile counterpart. There is only one thing you need to change: it needs to inherit from our custom DurableMossResourceManager base class. The next code listing shows the complete class: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;

namespace TxTest.MossTx.Durable
{
public class DurableFolderCommand : DurableMossResourceManager, IEnlistmentNotification
{
#region ctor
  public DurableFolderCommand(SPFolder objFolder)
  {
   Folder = objFolder;
  }
#endregion

  public void SetValue(string strKey, string strValue)
  {
   EnlistTransaction();

   if (Folder.Properties.Contains(strKey))
   {
    SaveOrgValue(strKey, Folder.Properties[strKey].ToString());
   }
   else
   {
    SaveOrgValue(strKey, String.Empty);
   }

   Folder.Properties[strKey] = strValue;
  }

  public void Update()
  {
   Folder.Update();
  }

  public override void Rollback(Enlistment enlistment)
  {
   foreach (string strKey in UndoLog.Keys)
   {
    Folder.Properties[strKey] = UndoLog[strKey];
   }

   Folder.Update();

   enlistment.Done();
  }

#region props
  private SPFolder _objFolder;
  public SPFolder Folder
  {
   get { return _objFolder; }
   set { _objFolder = value; }
  }
#endregion
}
}

As a point of interest, we will also use the DistributedTransactionStarted event of the TransactionManager object to keep us notified when the current transaction becomes a distributed transaction. To do this, we’ll need to define an event handler for this event, like so: TransactionManager.DistributedTransactionStarted += new TransactionStartedEventHandler(TransactionManager_DistributedTransactionStarted);

static void TransactionManager_DistributedTransactionStarted(object sender, TransactionEventArgs e)
{
Console.WriteLine("Transaction {0} became distributed, promoted from LTM to DTC",
Transaction.Current.TransactionInformation.DistributedIdentifier);
}

Once you start creating a durable resource, you can monitor the DTC (see section "Background info") and notice that a new transaction appears in the Transaction List. You will also notice that the DistributedTransactionStarted event is fired as soon as the first durable resource is enlisted. If you don’t like this, you might consider implementing a resource manager that supports the Single-Phase commit protocol. See section "Implementing a Single-Phase commit resource manager" for more information about that.

In the TransactionManager_DistributedTransactionStarted event handler you may have noticed that we’re outputting something called a distributed identifier. This value is filled once the transaction becomes distributed and can be used to map the current running transaction to the transactions you can monitor in the DTC.

The next code listing shows the part where a new distributed transaction started event handler is defined as well as the creation of a new transaction spanning multiple durable resource managers: TransactionManager.DistributedTransactionStarted += new
TransactionStartedEventHandler(TransactionManager_DistributedTransactionStarted);
using (TransactionScope ts = new TransactionScope())
{
objCommand6 = new DurableFileCommand(objFile1, objTransactionGuid);
objCommand6.CheckOut();
objCommand6.SetValue("ValueA", "value " + DateTime.Now);
objCommand6.SetValue("ValueA", "another value" + DateTime.Now);
objCommand6.SetValue("ValueB", "value of related docs" + DateTime.Now);
objCommand6.Update();
objCommand6.CheckIn("checked in file 1");

objCommand7 = new DurableFileCommand(objFile2, objTransactionGuid);
objCommand7.CheckOut();
objCommand7.SetValue("ValueA", "command 2 value " + DateTime.Now);
objCommand7.Update();
objCommand7.CheckIn("checked in file 2");

ts.Complete();
}

Recovery support

The main difference between volatile System.Transactions resource manager and durable System.Transactions resource managers is that a volatile System.Transactions resource manager doesn’t need recovery support at all and a durable System.Transactions resource manager should be able to recover after a failure. We won’t get to deep into this topic, but if you want to implement recovery support there are a couple of things you need to do.

First of all, you need to change your implementation of the Prepare() method. In this method, you need to save recovery info to some durable storage system (such as the file system, or a database). You may want to reread the section "Redo log" to check what kind of info you want to put in a redo log.

You will also need to change your implementation of the Commit() method. Since the System.Transactions resource manager finished it’s part of the job, you don’t need to recover anything and you don’t need a recovery log anymore. So, this is a good time to remove the recovery information concerning this particular transaction.

The Rollback() method needs to be changed as well in much the same way as the Commit() method. After successfully undoing any changes you won’t need a recovery log anymore, so this is a good place to remove the recovery information concerning this particular transaction.

Also, you need to call the Reenlist() method of the transaction coordinator and pass it a GUID that uniquely identifies a specific transaction and a specific System.Transactions resource manager. This GUID must be identical to the GUID you’ve used during the enlistment of the System.Transactions resource manager during the initial transaction. In our example, we’ve created a property called TransactionGuid that could be used for this purpose. If you’re planning to support a recovery process, you’ll need to persist such transaction GUIDs (something we didn’t do). If you’re interested in implementing recovery support, the following links may be of interest to you:

Implementing a Single-Phase commit resource manager

As you’ve seen in the section "Creating a durable resource manager", the transaction is promoted once the first durable resource manager is enlisted. As discussed in section "Background info", things don’t have to be like that. If you’d create a Single-phase commit resource manager it uses the LTM transaction coordinator until it really needs to promote a lightweight transaction to a distributed transaction. So let’s do exactly that...

Every Single-Phase commit resource manager needs to implement the ISinglePhaseNotification interface. This interface defines a single method called SinglePhaseCommit() that is called if the transaction is successful and isn’t promoted during its execution.

First off, we’ll create an abstract base class that implements that interface but leaves the implementation of the SinglePhaseCommit() method to its children. The main task of this abstract base class, which we will call SinglePhaseMossResourceManager, is to implement the enlistment process. To demonstrate the difference between the example described in section "Creating a durable resource manager" we’ll have participants enlist as durable resource managers. If you recall, in the previous example the transaction became a distributed transaction as soon as the first durable resource manager was enlisted. In this example, a transaction is only promoted if necessary. For instance, this might happen once the second durable resource manager is enlisted. The following code listing shows the enlist implementation of the abstract SinglePhaseMossResourceManager class. using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using TxTest.MossTx.TM;

namespace TxTest.MossTx.SinglePhase
{
public abstract class SinglePhaseMossResourceManager : ISinglePhaseNotification
{
#region ctor
  public SinglePhaseMossResourceManager()
  {
  }
#endregion

  public void EnlistTransaction()
  {
   if (IsTransactionEnlisted) return;

   Transaction tran = Transaction.Current;
   if (tran != null)
   {
    tran.EnlistDurable(TransactionGuid, this, EnlistmentOptions.None);
   }

   IsTransactionEnlisted = true;
  }

#region IEnlistmentNotification Members
  public void Commit(Enlistment enlistment)
  {
   enlistment.Done();
  }

  public void InDoubt(Enlistment enlistment)
  {
   // Do nothing.
  }

  public virtual void Prepare(PreparingEnlistment preparingEnlistment)
  {
   preparingEnlistment.Prepared();
  }

  public abstract void Rollback(Enlistment enlistment);
  public abstract void SinglePhaseCommit(SinglePhaseEnlistment enlistment);
#endregion

  protected void SaveOrgValue(string strKey, string strOldValue)
  {
   if (!UndoLog.ContainsKey(strKey))
   {
    UndoLog.Add(strKey, strOldValue);
   }
  }

#region props
  private bool _blnIsTransactionEnlisted;
  public bool IsTransactionEnlisted
  {
   get { return _blnIsTransactionEnlisted; }
   set { _blnIsTransactionEnlisted = value; }
  }

  private bool _blnMetadataIsDirty;
  public bool MetadataIsDirty
  {
   get { return _blnMetadataIsDirty; }
   set { _blnMetadataIsDirty = value; }
  }

  private Dictionary _objUndoLog = new Dictionary();   public Dictionary UndoLog
  {
   get { return _objUndoLog; }
   set { _objUndoLog = value; }
  }

  private Guid _objTransactionGuid = Guid.NewGuid();
  public Guid TransactionGuid
  {
   get { return _objTransactionGuid; }
   set { _objTransactionGuid = value; }
  }
#endregion
}
}

That leaves the implementation of the SinglePhaseCommit() method to its children, which will let the transaction coordinator know that the commit was successful. The rest of the implementations of the child classes isn’t that different from what we’ve seen before, so we won’t discuss it explicitly. Without further ado, here’s the implementation of the SinglePhaseFileCommand: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;
using TxTest.MossTx.TM;

namespace TxTest.MossTx.SinglePhase
{
public class SinglePhaseFileCommand : SinglePhaseMossResourceManager, ISinglePhaseNotification
{
#region ctor
  public SinglePhaseFileCommand(SPFile objFile)
  {
   File = objFile;
  }
#endregion

  public override void SinglePhaseCommit(SinglePhaseEnlistment enlistment)
  {
   enlistment.Done();
   //enlistment.Committed();
  }

  public void SetValue(string strKey, string strValue)
  {
   EnlistTransaction();
   SaveOrgValue(strKey, File.Properties[strKey].ToString());
   File.Properties[strKey] = strValue;
  }

  public void CheckOut()
  {
   File.CheckOut();
  }

  public void CheckIn(string strComment)
  {
   File.CheckIn(strComment);
  }

  public void Update()
  {
   File.Update();
  }

  public override void Rollback(Enlistment enlistment)
  {
   if (File.CheckOutStatus == SPFile.SPCheckOutStatus.None)
   {
    File.CheckOut();
   }

   foreach (string strKey in UndoLog.Keys)
   {
    File.Properties[strKey] = UndoLog[strKey];
   }

   File.Update();
   File.CheckIn("rollback because of a failed transaction");

   //enlistment.Done();
  }

#region props
  private SPFile _objFile;
  public SPFile File
  {
   get { return _objFile; }
   set { _objFile = value; }
  }
#endregion
}
}

The Single-Phase commit variant of the Folder command looks like this: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using Microsoft.SharePoint;
using TxTest.MossTx.TM;

namespace TxTest.MossTx.SinglePhase
{
public class SinglePhaseFolderCommand : SinglePhaseMossResourceManager, ISinglePhaseNotification
{
#region ctor
  public SinglePhaseFolderCommand(SPFolder objFolder)
  {
   Folder = objFolder;
  }
#endregion

  public override void SinglePhaseCommit(SinglePhaseEnlistment enlistment)
  {
   enlistment.Committed();
  }

  public void SetValue(string strKey, string strValue)
  {
   EnlistTransaction();

   if (Folder.Properties.Contains(strKey))
   {
    SaveOrgValue(strKey, Folder.Properties[strKey].ToString());
   }
   else
   {
    SaveOrgValue(strKey, String.Empty);
   }

   Folder.Properties[strKey] = strValue;
  }

  public void Update()
  {
   Folder.Update();
  }

  public override void Rollback(Enlistment enlistment)
  {
   foreach (string strKey in UndoLog.Keys)
   {
    Folder.Properties[strKey] = UndoLog[strKey];
   }

   Folder.Update();

   enlistment.Done();
  }

#region props
  private SPFolder _objFolder;
  public SPFolder Folder
  {
   get { return _objFolder; }
   set { _objFolder = value; }
  }
#endregion
}
}

And the client that starts the transaction goes something like this: using (TransactionScope ts = new TransactionScope())
{
objCommand9 = new SinglePhaseFileCommand(objFile1);
objCommand9.CheckOut();
objCommand9.SetValue("Dossiernummer", "value " + DateTime.Now);
objCommand9.Update();
objCommand9.CheckIn("test");

objCommand10 = new SinglePhaseFileCommand(objFile2);
objCommand10.CheckOut();
objCommand10.SetValue("Dossiernummer", "value " + DateTime.Now);
objCommand10.Update();
objCommand10.CheckIn("test");

ts.Complete();
}

If you’d check the TransactionManager_DistributedTransactionStarted event handler (discussed in section "Creating a durable resource manager"), you’ll notice that the enlistment of the first durable resource manager does not result in the creation of a distributed transaction... yet. This happens as soon as the second durable resource manager is enlisted which causes the transaction to be promoted. If, instead of enlisting two durable resource managers you would have enlisted only one, you would have seen the Single-Phase commit protocol in action. In such a scenario, when the transaction is completed the SinglePhaseCommit() method is called. In the current scenario we’re dealing with a distributed transaction, which, when the transaction completes, results in the calling of the 2PC methods Prepared() and Commit().

Implementing a custom transaction manager

It’s possible to create a custom transaction manager. In all likelihood, you will not do this as you can already use the LTM and OleTx transaction coordinators, but it would be nice to take a look at how one would accomplish this. In this section, we’re discussing a simple custom transaction manager.

First of all, such a custom transaction manager needs to implement the IPromotableSinglePhaseNotification interface (see section "IPromotableSinglePhaseNotification interface" for further details). In this example, we won’t bother to implement the Initialize() method that is used to notify transaction participants that enlistment has been completed. We’ll implement a very basic version of the Rollback() method that indicates the transaction is aborted. The implementation of the SinglePhaseCommit() method is also very simple; it just indicates the transaction is committed. These implementations look like this: public void Initialize()
{
//
}

public void Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
{
singlePhaseEnlistment.Aborted();
}

public void SinglePhaseCommit(SinglePhaseEnlistment singlePhaseEnlistment)
{
singlePhaseEnlistment.Committed();
}

You’ll also need to implement the Promote() method that returns a propagation token when the transaction handled by the custom System.Transactions resource manager is promoted. You can call the GetTransmitterPropagationToken() method of the TransactionInterop class to accomplish this, like so: public byte[] Promote()
{
return TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
}

In case our custom System.Transactions resource manager feels the need to communicate with its transaction participants, we’ll also add a collection of such participants. This is shown in the next code listing: public void AddResourceManager(ISinglePhaseNotification objResourceManager)
{
_objResManagers.Add(objResourceManager);
}

#region prop
private List _objResManagers = new List();
public List ResourceManagers
{
get { return _objResManagers; }
set { _objResManagers = value; }
}
#endregion

The complete implementation of our custom System.Transactions resource manager looks like this: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;

namespace TxTest.MossTx.TM
{
public class CustomTransactionManager : IPromotableSinglePhaseNotification
{
  public CustomTransactionManager()
  {
  }

  public void Enlist()
  {
  }

#region IPromotableSinglePhaseNotification Members
  public void Initialize()
  {
   //
  }

  public void Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
  {
   singlePhaseEnlistment.Aborted();
  }

  public void SinglePhaseCommit(SinglePhaseEnlistment singlePhaseEnlistment)
  {
   singlePhaseEnlistment.Committed();
  }
#endregion

#region ITransactionPromoter Members
  public byte[] Promote()
  {
   return TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
  }
#endregion

  public void AddResourceManager(ISinglePhaseNotification objResourceManager)
  {
   _objResManagers.Add(objResourceManager);
  }

#region prop
  private List<ISinglePhaseNotification> _objResManagers = new List<ISinglePhaseNotification>();
  public List<ISinglePhaseNotification> ResourceManagers
  {
   get { return _objResManagers; }
   set { _objResManagers = value; }
  }
#endregion
}
}

We will use the Single-Phase commit System.Transactions resource managers we’ve created earlier in combination with our custom transaction coordinator. To support this, we have to change the enlistment process a little bit. The enlistment process is implemented in the EnlistTransaction() method of the SinglePhaseMossResourceManager class and needs to take care of two things:

Let the current transaction know it will be handled by our custom transaction coordinator.
Add the transaction participant (our Single-Phase commit System.Transactions resource manager) to the list of participant members of our custom transaction coordinator.

The following code listing shows the enlistment process: public void EnlistTransaction()
{
if (IsTransactionEnlisted) return;

Transaction tran = Transaction.Current;
if (tran != null)
{
tran.EnlistPromotableSinglePhase(CurrentTransactionManager
}

IsTransactionEnlisted = true;
}

This results in the following implementation of the SinglePhaseMossResourceManager class: using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Transactions;
using TxTest.MossTx.TM;

namespace TxTest.MossTx.SinglePhase
{
public abstract class SinglePhaseMossResourceManager : ISinglePhaseNotification
{
#region ctor
  public SinglePhaseMossResourceManager(CustomTransactionManager objCurrentTransactionManager, Guid objTransactionGuid)
  {
   CurrentTransactionManager = objCurrentTransactionManager;
   TransactionGuid = objTransactionGuid;
  }
#endregion

  public void EnlistTransaction()
  {
   if (IsTransactionEnlisted) return;

   Transaction tran = Transaction.Current;
   if (tran != null)
   {
    tran.EnlistPromotableSinglePhase(CurrentTransactionManager);
    CurrentTransactionManager.AddResourceManager(this);
   }

   IsTransactionEnlisted = true;
  }

#region IEnlistmentNotification Members
  public void Commit(Enlistment enlistment)
  {
   //enlistment.Done();
  }

  public void InDoubt(Enlistment enlistment)
  {
   // Do nothing.
  }

  public virtual void Prepare(PreparingEnlistment preparingEnlistment)
  {
   preparingEnlistment.Prepared();
  }

  public abstract void Rollback(Enlistment enlistment);
  public abstract void SinglePhaseCommit(SinglePhaseEnlistment enlistment);
#endregion

  protected void SaveOrgValue(string strKey, string strOldValue)
  {
   if (!UndoLog.ContainsKey(strKey))
   {
    UndoLog.Add(strKey, strOldValue);
   }
  }

#region props
  private bool _blnIsTransactionEnlisted;
  public bool IsTransactionEnlisted
  {
   get { return _blnIsTransactionEnlisted; }
   set { _blnIsTransactionEnlisted = value; }
  }

  private bool _blnMetadataIsDirty;
  public bool MetadataIsDirty
  {
   get { return _blnMetadataIsDirty; }
   set { _blnMetadataIsDirty = value; }
  }

  private Dictionary _objUndoLog = new Dictionary();
  public Dictionary UndoLog
  {
   get { return _objUndoLog; }
   set { _objUndoLog = value; }
  }

  private CustomTransactionManager _objCurrentTransactionManager;
  public CustomTransactionManager CurrentTransactionManager
  {
   get { return _objCurrentTransactionManager; }
   set { _objCurrentTransactionManager = value; }
  }

  private Guid _objTransactionGuid;
  public Guid TransactionGuid
  {
   get { return _objTransactionGuid; }
   set { _objTransactionGuid = value; }
  }
#endregion
}
}

The client code looks like this: TransactionManager.DistributedTransactionStarted += new
TransactionStartedEventHandler(TransactionManager_DistributedTransactionStarted);
using (TransactionScope ts = new TransactionScope())
{
CustomTransactionManager objTM = new CustomTransactionManager();

objCommand9 = new SinglePhaseFileCommand(objFile1, objTM, objTransactionGuid);
objCommand9.CheckOut();
objCommand9.SetValue("KeyA", "value " + DateTime.Now);
objCommand9.Update();
objCommand9.CheckIn("test");

objCommand10 = new SinglePhaseFileCommand(objFile2, objTM, objTransactionGuid);
objCommand10.CheckOut();
objCommand10.SetValue("KeyA", "value " + DateTime.Now);
objCommand10.Update();
objCommand10.CheckIn("test");

ts.Complete();
}

This concludes our visit of an implementation of a basic custom transaction coordinator.

Conclusion

The SharePoint 2007 framework lacks transaction support when dealing with list items, especially a transaction mechanism that can be used in conjunction with the .NET System.Transactions namespace. There are situations when this is very inconvenient, even when dealing with semi-structured information. We hope this is something that’ll change in the future, and in the mean time, you’ll have to create a System.Transactions resource manager yourself to be able to deal with scenario’s that require transactions in MOSS. Creating a System.Transactions resource manager turns out to be not that difficult, understanding most aspects that are involved can get pretty complex. If you’re planning on walking the System.Transactions road to SharePoint, this article will get you a long way.

More information

If you want more information about related topics discussed in this article, you could check out the following links:

Working with large files

10 June 2008 07:05:32

Working with large files

One of our customers had issues working with Microsoft Office SharePoint Server 2007 and large files. They received time-out errors as well as general errors when working with large files (in this case, files that are larger than 25 MB). A small percentage of the files they work with are really large; they are close to 150MB in size. After trying out several solutions in a Knowledge Base article and Joel Oleson’s blog post "File Name, Length, Size and Invalid Character Restrictions and Recommendations" we were still not very happy with the results. At that point, we decided to call MS Support to try and find out if what we were trying to achieve was even feasible. This resulted in a lot of useful information about working with large files in MOSS. Mind you, we did not come up with most of this info ourselves, we have to thank MS Support and MS documentation for that (and most specifically Denes Theisz of MS Support), but we decided to blog about it anyway because we’re convinced it contains valuable information that can help out other people as well.

Average file sizes

First of all, what qualifies as a large file? The default file upload size is 50MB. This limit applies to a single item, although it also seems to equate to the max upload size of the entire batch of files that is uploaded via the Upload > Upload Multiple Documents option in the user interface of a list.

In his (very interesting and useful) blog post “File Name, Length, Size and Invalid Character Restrictions and Recommendations”, Joel Oleson mentions the default max file size of 50 MB is also the one that is recommended for the best experience, that Microsoft IT limits the max file upload size to 100 MB, that large files of 200-500 MB could be supported in a LAN, and that the absolute limit of the max file size is 2GB (which is a limit imposed by SQL Server). All of these numbers are, at least in our experience, quite optimistic, but we’ll get back to that later.

Let’s quote Joel Oleson one more time... Joel has also created an interesting presentation called "Is the File Server dead?" that expresses that MOSS shouldn’t be used as a substitute for the Windows file system. In this presentation, he describes a sampling of files that can be found in a SharePoint farm that contains around 2.5 million files (which equates to a size of roughly 2 TB). After analyzing the contents of this farm, Joel Oleson concludes that most files used in that MOSS environment consist of Office files (which we assume will be true for most MOSS implementations, although PDF files will typically make up a large part of the contents as well), 32% of them being .doc files with average file sizes of 500KB. In this specific farm, the largest files are up to 5MB.

These findings match our own experiences, accept for cases in which a company scans content and places that content in MOSS or works with large drawings. In such cases, file sizes exceed these numbers considerably. In such scenarios, a document size that lies somewhere between 100MB and 150MB is certainly possible.

How to facilitate working with large files?

All in all, working with large files in a MOSS environment is less than ideal. Having said that, there are a couple of things you can do to make working with large files easier.

Network bandwidth

First of all, you need to optimize the available network bandwidth. If you’re planning to work with large files in MOSS, your network bandwidth should be at least 100 Mbit full duplex. You can use the MOSS Multiple Document Upload tool to provide an estimate of the network speed currently available to you (although this estimation in general seems to be too pessimistic). There are two problems associated to having a slow network:

It takes a while before the end user is able to open or save a document (this is an obvious one).
The document (or at least parts of the document) gets loaded in the memory of the MOSS web front end (WFE) handling the request, and the SQL Server containing the document. The longer it takes the end user to open a document, the longer server resources are tied up. Memory usage suffers the most from this. So, please note, having a slow network can have a severe impact on the performance of your MOSS farm.

SharePoint configurations and installation

There are a couple of SharePoint configurations that make it easier to work with large files in MOSS. They are:

Adjust the maximum upload size in SharePoint Central Administration (SCA).
Adjust the IIS connection time-out settings.
Adjust various ASP.NET time-out settings for a specific SharePoint web application.
Adjust the default chunk size.
Use 64-bit web front-ends.

All the configuration changes are described in detail in the KB article and Oleson’s blog post mentioned earlier in this article, so there’s no need to discuss the how-to’s in this article all over again.

Chunk size bug in MOSS

The chunk size determines the amount of data that the client retrieves in one go when opening a document. For example, if a client tries to open a document of 50MB and the chunk size is 10MB, the document is divided and retrieved in 5 chunks. Each chunk will be loaded into the memory of both the WFE handling the request and SQL Server. The default chunk size is 5MB and you can adjust the chunk size by issuing the following stsadm command:

Stsadm.exe -o setproperty -pn large-file-chunk-size -pv [size in bytes]

As you’ve seen, you can change the chunk size setting. Raising it could benefit the client because a big chunk of 50MB loads faster than 10 chunks of 5MB. Then again, this also means that 50MB gets loaded into the memory of the WFE and SQL Server (instead of only 5MB at a time), thus increasing the load on your MOSS farm. This memory will only be released when the client has finished loading the entire document. It depends on your farm and the number of users accessing it if it’s a viable alternative for you to raise the chunk size. You can only answer this question truthfully by trying and monitoring your server farm intensively.

Now, let’s discuss the problem associated to chunk size. Because of a bug in MOSS, the chunk size doesn’t work within Explorer view (or any other method that uses WebDAV as the underlying protocol). This is a bug that only has been discovered recently. Currently there is nothing you can do about this problem, MS expects a bug fix rollup that is currently scheduled around August 2008 that will address the chunk size bug.

As a result, opening a file of 50MB via Explorer view places a much higher load on the server ( 50MB is loaded into the memory of the WFE for this document) compared to performing the same action via the standard SharePoint user interface (default, max 5 MB is loaded into the memory for this document).

32-bit versus 64-bit WFEs

MS strongly advises to install 64-bit installations of MOSS on WFEs, unless you have a very good reason not to do this (this advise can be found in the MS best practices document "Planning and Deploying Service Pack 1 for MOSS 2007 in a multi-server environment").

As discussed, if you use WebDAV, chunking is ignored. In such cases, accessing a large file uses a lot of the available memory on the WFE. On a 32 bit WFE, the W3WP.exe process hosting a SharePoint web application will not be able to consume more than around 800MB memory. After that, you run the risk that the process is restarted automatically, thus crashing the worker process and resulting in the failure of the download. It is easy to see that multiple users working with large files can lead to lots of worker process recycles, thus potentially obliterating server performance and end user experience.

By the way, as a point of interest, directly after such a crash occurs, you actually have a higher chance of successfully downloading the document, as the freshly restarted worker process will typically have enough memory to spare.

64 bit WFEs hold up better when working with large files, because the worker processes won’t recycle, but will only use a lot of memory (and eventually the swap file will be used a lot). Of course, this could lead to a disk bottleneck and slow performance, but we prefer this situation over the other one.

Client settings

You’ve seen that there a couple of steps you can take to improve the ability to work with large files in a MOSS environment by changing the server environment. There are also some steps that need to be taken on the client. We’ll discuss them in this section.

Make sure that the size of the Internet Explorer cache (located in the Temporary Internet Files folder) of end users has a size of at least 50 MB. End users that regularly work with large files should have a cache size that is even larger. Failing to do so will increase the load on the WFEs considerably.

End users working with large files (files that are 25MB+ in size) should first save them locally (btw, Office 2007 makes this a lot easier), edit them and then upload them again to MOSS. Failing to do so increases the network traffic and memory usage on the WFEs.

Download a document by right-clicking it and choosing Save Target As (Windows functionality) instead of Send To > Download a copy (MOSS functionality). The first one performs better, since the latter one causes the complete document to be loaded in the memory of the WFE. By the way, uploading large files works best and fastest using the Explorer view.

End users shouldn’t use the Explorer view when browsing files and folders. Placing the mouse cursor over a file or folder within the Explorer view leads to the retrieval of all metadata of all files in the folder that is currently viewed. In some scenarios, browsing files via the Explorer view might even lead to the full retrieval of those files. Failing to avoid the Explorer view when browsing files and folders increases the server load considerably. Instead, you should use a SharePoint view (for instance, the All Documents view) for browsing.

As a general rule, if possible, when using the SharePoint user interface, don’t work with files larger than 25MB. When working with WebDAV (i.e. via a file share), don’t work with files larger than 10MB. Please note that this info is only based on tests performed on a couple of test environments, so you might find different results yourself. Having said that, we do believe most environments will find that end users shouldn’t work with files in MOSS that are much larger than the sizes mentioned.

Large files that are often read by end users, but seldom updated, should be kept outside MOSS and stored on a file share that is indexed by MOSS.

Monitor working with large files

If you plan to work large files, monitoring your MOSS environment should be a part of your implementation strategy. In this section, we’ll discuss what we’ve done to gain insight in our server farm when end users start working with large files.

Please note: In our analysis, we’ve focused primarily on the WFEs, since we had indications that the bottleneck was to be found there. In most cases, you should also take an extensive look at the SQL Server acting as the data repository in your MOSS farm.

Analyzing performance counters

First of all, we’ve used perfmon and created a counter log for the following performance objects:

LogicalDisk
Memory
Network interface
Paging file
PhysicalDisk
Process
Processor

Use a sample interval of 5 seconds and save the output to a binary file. Then, start perfmon manually and simulate one (or more clients) working with large files. When finished, you can open the binary file at a later time using one of the following methods:

Open a command prompt and type perfmon. This opens the Performance dialog window.
In the toolbar, click the View Log Data (Ctrl+L, the fourth icon on the left, the one that looks like a database) button. This opens the System Monitor Properties window.
In the Data source section, click the Log files radio button.
Click the Add button and locate the blg file containing the performance counters you're interested in.
Click OK.
Click the Add button (Ctrl+I, the + sign).
Make sure the Select counters from computer radio button is selected and choose the server you're interested in.

After that, you're ready to analyse any performance counter you want. Alternatively, you can convert the binary log file to a .cvs file and analyse that. The next procedure explains how to do this:

Open a command prompt.
Navigate to the folder containing the binary log file (.blg).
Type: relog [name log file].blg -f CSV -o [name csv file].csv

At this point, you could, for instance, open the .csv file in Excel.

When you’re ready the analyze the performance counters, you should at least consider to take a look at the following information:

LogicalDisk

Check the Current Disk Queue performance counter. If it’s low, system requests don’t have to wait for disk access (for instance, for reading and writing files), indicating that the hard disk is fast enough. You can also use the LogicalDisk performance object to determine if there is enough free space left on the server hard disks.

Memory

Microsoft Office SharePoint Server 2007 servers need memory – and lots of it. Use the MS Capacity Planner tool if you need specific guidelines. In general, a WFE needs at least 4GB. You should take a look at the Available Mbytes performance counters to obtain specifics.

Network interface

If you want to work with Microsoft Office SharePoint Server 2007 and large files, you need to make sure that the server is able to process all the network traffic. Take a look at the Bytes Total/sec counter. You should also take a look at the Output Queue Length which contains outbound packets and should remain below 2. You could also check the Packed Received Errors counter to determine if any network errors have occurred. If there is a problem, you should replace the current network adapter with a faster one, or add a new one.

Paging file

Check the % Usage counter. If it’s high, this might mean a lot of swapping is going on, which can be very detrimental to the performance of your server. In such cases, you need to add more memory.

Processor

Look at the % Processor Time counter to establish how busy the server processor is. Various guidelines exist, we use the rule that if you’re using 75% or more of the processor time, consistently, things become critical and your server is too busy. In such cases, you might replace the processor with a faster one, add new processors or implement some kind of scale-out strategy.

Process

Microsoft Office SharePoint Server 2007 web applications are hosted within their own application pools, which are eventually hosted within their own worker process (an instance of w3wp.exe). So, you should definitely take a look at the % Processor Time corresponding to the various instances of w3wp.exe on your server. Also, take a look at the Private bytes counter to see how much memory is allocated per worker process. You can expect to see this number steadily climbing when the server has more difficulty processing end user requests for large files. Eventually, such behavior can result (on 32bit WFEs) in automatic recycles of the w3wp processes.

In addition to keeping track of performance counters, you might want to open a command prompt and execute iisapp.vbs to gain insight in the various application pools that run on the WFE (and the w3wp process instances that host the application pools). At a later stage, we will look at Process performance counters that are related to instances of w3wp.exe processes that host SharePoint web applications, so it might be useful to have more knowledge about the available app pools, although unfortunately you have to realize that you can’t use the output of iisapp.vbs to map a specific application pool to a specific process performance counter of a w3wp process later. Btw, we have yet to find a way to do this automatically.

Other interesting log files

To get a complete picture of the status of your server, you should also collect and analyze the following:

Event viewer application log
Event viewer system log
IIS log for the MOSS web application
HTTPERR log (if it exists)
ULS (the LOGS folder in the 12 hive).

Conclusion

Of course, when working with large files and Microsoft Office SharePoint Server 2007, the speed of your network is very important. It may be less obvious that a slow network can also have a severe impact on the performance of your WFEs as well as the SQL Server containing your content databases. In addition, MOSS isn’t very suitable to work with large files (files larger than 25MB) and is most suitable to work with Office files (which are seldomly larger than 5MB). Be careful when working with the Explorer view, it might tax your servers more than you realize (although this will be addressed in a bug fix that has yet to be released).

If you want to work with large files in a MOSS environments, there are various factors you can optimize:

Optimize network speed.
Use 64-bit web front-ends.
If you’re using 32-bit WFEs, you can influence the recycle behaviour (http://support.microsoft.com/kb/332088 ).
Try to avoid to work with files larger than 25MB (this may be considerably less when using WebDAV) .
Try to store large files on a file share (not in MOSS) and have them indexed by MOSS.
Optimize the chunk size, but realize that this (because of a bug) currently does not influence anything that uses WebDAV (such as the Explorer View).
Optimize client settings.
Teach end users how to work with MOSS, thus preventing expensive actions (such as browsing for files using the Explorer View).
Install the August 2008 (or whenever it will actually be released) bug fix rollup as soon as you can.

In this section, we’ve summarized the lessons we’ve learned so far when it comes to working with large files in a MOSS environment. Our conclusion is the following: if you want to do it, work lies ahead of you!