Findability is Overrated

There is such a focus when planning an enterprise search or information management deployment that they’ll be doomed if they don’t plan for findability. I’ve seen these crazy stats quoted even to justify the effort to put in to this planning – crazy claims like one I saw from IBM stating the average information worker spends an average of two hours per day looking for information. Wow, can you imagine having workers like this on your team spending a couple hours every day just looking for things? Aside from researchers, who in their right mind would spend an average of a quarter of their eight-hour day lost and looking for things? Sure, there’s a lot of information out there, but what I work with most frequently I should hope I remember where I keep it or store it in a handy place so I don’t have to spend two hours searching for it tomorrow.

Findability is overrated. It is the idea that you need to think of all the ways that users might search for a given piece of content so that you can capture all the right metadata and store it in the right place. And this may seem logical, given how these crazy stats get quoted that apparently the average worker spends about a quarter of their workday looking for stuff. That’s a lot of productivity lost! Of course one would naturally want to make all that apparent searching more productive and help their users find all that content. It leads to this notion of making information more findable: if you put enough planning around the information architecture, there will be an explosion of productivity as users free up upwards of 25% of their workday from having to search for all this information.

It’s ludicrous, somewhat. I like the idea of connecting users to information, and improving that overall process. But instead of just making a faster widget, is there a way to enhance the whole process to get that desired end result? If it’s about connecting users to information, that can be a whole lot of things beyond one blinder-focused solution built around an anticipated search query. Besides, relevancy in searches is personal and in context, not simply preparing for what is considered a popular top query in any context.

What if instead of focusing on all the ways users might want to search for a given piece of content, the focus rather was on what content a given user might need at different times? Instead of the findability of content, let the focus be on the discoverability of content. A lot of the exercises and design may be the same: you still need to tag and organize content; the difference is the concentration in how the content will be used, namely what value it brings. Information discovery: how can the right information be exposed to the right user in the right context, at just the right time? (Maybe even without the user having to go searching for it).

I love Amazon, because they expose the right information to me in the right context, and usually at just the right time. When I open the home page, they show me different books I may be interested in, which often I am, and this satisfies me right away. Other times, there’s a book I’m already thinking about and do a quick search to find. While I’m reading a summary about this book, Amazon also suggests other similar books I may be interested in, and if I add it to my cart I get presented with other books that are likely to be of interest related to what I was shopping for during this and recent sessions.

Telemarketers use this technique as well when the caller’s computer updates their script depending on how the call is going. My account manager at my bank has offers on his computer prompting him to offer me different products and services at different times, depending on the context of what he is reviewing (although I always find it a little weird when I go in to withdrawal a roll of dollar coins to have change handy for parking and laundry, and this somehow leads to a discussion on whether I’m interested in a new mortgage).

Some companies and industries are well on their way with this focus. It’s no different really than having efficiency experts map the process on a factory floor around what adds value to tasks the workers actually perform, and bringing that concept to understanding what the knowledge worker does and what would be valuable information to make available to them, in what context, and at what time. The underlining technology is still search driven; the focus is just on the user’s experience and anticipated value interacting with the content: a focus on information discoverability rather than simply generic findability of a piece of content.

What IT Projects Could Learn from Medical Procedures

A few weeks ago I had a medical procedure, after which oddly stimulated some thought on the parallels between delivering an IT project and conducting a medical procedure. Everything was a success without any real complications, and I’m a little healthier now because of it: all great outcomes! But there was some unexpectedness or moments where the doctor had to adapt to new information discovered during the procedure – or perhaps, these were refined requirements in IT project speak. What were some of these characteristics of their process that seemed to set the doctor and her team up for success that often might be lacking in an IT project delivery? Let me share one with you about the shutdown point.

It turned out I had an infection brewing inside me just below my shoulder on my back, which probably explains why for much of this summer I had felt quite run down physically and my concentration didn’t feel as sharp as usual. Somehow one of my oil glands had got clogged, backed up, and this caused an infection to develop. It’s nothing too serious, I don’t think, but the doctor said she would have to cut in to drain and clean the infected area, and would go ahead and perform the procedure in a few minutes. What a treat: some nice surprise Saturday afternoon surgery!

After planning what she needed and freezing the area to prep, she made an incision. This was probably quite a routine process for her, so I was mostly focused on just waiting for her to finish. Here’s where the weird part happened: suddenly she reports to me that she isn’t finding the affected area and is going to close it back up and stop the procedure. Wait, what?

There I was lying there, with a couple inch cut in my back; I felt pretty committed to completing the task obviously, and she was committed until all of the sudden it was time to stop. Then I hear her say she found it, that she had cut a little deeper while she was deciding to halt the procedure and had found the infection; it was just deeper than she had expected. This really stuck out for me: she was ready to shut it all down and take a step back when she got a look under the skin and saw it may be bigger or deeper than she originally assessed. Stopping the procedure wouldn’t be a failure; in fact she said it like it was always an acceptable option that was perfectly okay or not at all unexpected.

Now in contrast, I’ve been on many IT projects, and these types of projects are hardly ever halted, no matter what the outlook for success looks like. There seems to be a stigma and unrealistic expectation that it’s not acceptable to halt an IT project, that doing so would mean there was a failure. Everyone from the project sponsor on down to the delivery team are conditioned with some dysfunctional commitment to seeing the project through, no matter what. I’ve been on several projects where I’ve sensed a disaster, yet even after raising a flag everyone still opts to stay the course. In September’s Harvard Business Review Magazine, there is an article that discusses exactly this, and they indicate that 67% of managers won’t shut down an IT project for their group, even when they know it is not likely to succeed.

It baffles me why I’ve known so many customers who decide to continue rather than halt a project, even when they know they have the wrong team or the project is heading way off course or the consultants they’ve hired seem to be coming up short. I wonder, is there a stigma that would make it feel like a failure if the project is simply shut down, so it is better to continue in hopes that somehow, magically, it will get back on track and recover? Or is the root cause that the procurement process is too difficult and adds too much overhead that projects tend to get defined by large generic contracts rather than contracting well to break it down into modules or distinct phases with a well-defined vision and clear expectations set?

Is it the existence of a contract itself that creates an environment of opposing sides to some degree that each have to remain mindful of protecting each of their own interests? Why would it be so easy and acceptable for a surgeon to end even a simple surgery before it achieves its original objective when things don’t turn out exactly as expected once they get a look below the skin, yet not so for an IT project? Perhaps contracts need a shutdown clause that defines the shutdown point up front before the project even initiates. I had the sense my doctor had a shutdown point in mind before she began the procedure, and this was why she was so prepared to stop and back out if something wasn’t as expected.

A healthy functional IT project considers a shutdown point before the project even kicks-off, and then checks in on that shutdown point regularly during status meetings to honestly assess how the project is trending in relation to when it should be halted. No more “Don’t ask; don’t tell” where there’s this unspoken commitment to continue no matter what, where the project blindly marches on to inadequacy and disappointment. Instead: regular checks against a perfectly acceptable shutdown point.

SharePoint Health Rules: My Sample Rule Catches Lazy Customization Deployments

A feature I was happy to see added to SharePoint 2010 was the health rules. Administrators can be tipped off when something is not quite right with a farm. Although there are a fantastic number of rules already defined and built in, I really like how easy and straight forward it is to create custom rules. The SPHealthAnalysisRule class has a Check method that is available to simply override and implement your own logic.

One thing I use to do when I looked after a number of farms was to run a number of scripts on the farm that assessed the health, evaluated usage, or the like. By reworking these scripts into health rules, they can run and report the health of the farm on a regular schedule. One example I’ve included below is a check that determines if any features have been deployed through a copy & paste of the feature files on the servers rather than through the preferred method of packaging them into a SharePoint solution package.

Other uses for custom health rules can include things like checking the lifecycle of a site and notifying with an informative list of sites that appear to have gone dormant. Or even for non-administrator health related checks, a custom health rule can check patterns of usage or inappropriate usage, monitor repositories, check if ex-employees are the sole owners of content, or maybe if content is persisting for long periods of time yet haven’t been declared a record. It is exciting all the different uses of a custom health rule you can use to poll your SharePoint farm for regular information or administrator warnings.

The following code illustrates how I created my health rule that checks for SharePoint features deployed on the server that are not packaged in a SharePoint solution package (WSP file). I list the built in features in a separate file I create, builtinfeatures.txt, and store in the Config directory in the SharePoint Root. Finally, I have included the feature activating and deactivating code to add/remove these health rules to the SharePoint farm.

 

using System;
using System.Collections.Generic;
using System.IO;
using System.Globalization;
using System.Text;
using Microsoft.SharePoint.Utilities;
using Microsoft.SharePoint.Administration;
using Microsoft.SharePoint.Administration.Health;
namespace Develops.SharePoint.HealthRule
{
    public class UnpackagedFeatureRule : SPHealthAnalysisRule
    {
        List<string> unpackagedFeatures = new
				List<string>();
        #region Properties
        public override string Summary
        {
            get { return "One or more custom features was deployed without packaging it in a SharePoint Solution Package."; }
        }
        public override string Explanation
        {
            get
            {
                StringBuilder builder = new StringBuilder();
                foreach (string feature in unpackagedFeatures)
                {
                    builder.AppendLine(feature);
                }
                return string.Concat(
                    "The following custom features were not associated with a SharePoint Solution Package: ", builder);
            }
        }
        public override string Remedy
        {
            get { return "Package the identified Features into SharePoint solution packages."; }
        }
        public override SPHealthCategory Category
        {
            get { return SPHealthCategory.Configuration; }
        }
        public override SPHealthCheckErrorLevel ErrorLevel
        {
            get { return SPHealthCheckErrorLevel.Warning; }
        }
        public override SPHealthAnalysisRuleAutomaticExecutionParameters AutomaticExecutionParameters
        {
            get
            {
                SPHealthAnalysisRuleAutomaticExecutionParameters retval = new SPHealthAnalysisRuleAutomaticExecutionParameters();
                retval.Schedule = SPHealthCheckSchedule.Monthly;
                retval.Scope = SPHealthCheckScope.All;
                retval.ServiceType = typeof(SPTimerService);
                return retval;
            }
        }
        #endregion
        #region Check
        public override SPHealthCheckStatus Check()
        {
            if (!SPFarm.Joined)
                throw new InvalidOperationException();
            List<string> systemFeatures = new List<string>();
            List<string> customFeatures = new List<string>();
            FileStream stream = new FileStream(
                SPUtility.GetGenericSetupPath(@"CONFIG") + @"\builtinfeatures.txt",
                FileMode.Open, FileAccess.Read);
            StreamReader reader = new StreamReader(stream);
            string line = reader.ReadLine();
            while (line != null)
            {
                systemFeatures.Add(line.ToLower(CultureInfo.CurrentCulture));
                line = reader.ReadLine();
            }
            reader.Dispose();
            stream.Dispose();
            SPFarm farm = SPFarm.Local;
            foreach (SPFeatureDefinition featureDef in farm.FeatureDefinitions)
            {
                if (featureDef.SolutionId.Equals(Guid.Empty) &&
                    !systemFeatures.Contains(featureDef.DisplayName.ToLower(CultureInfo.CurrentCulture)))
                {
                    unpackagedFeatures.Add(featureDef.DisplayName);
                }
            }
            if (unpackagedFeatures.Count == 0)
                return
				SPHealthCheckStatus.Passed;
            else
                return SPHealthCheckStatus.Failed;
        }
        #endregion
    }
}
using System;
using System.Collections.Generic;
using System.Reflection;
using System.Runtime.InteropServices;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration.Health;
namespace Develops.SharePoint.HealthRule.Features.HealthRules
{
    public class HealthRulesEventReceiver : SPFeatureReceiver
    {
        public override void FeatureActivated(SPFeatureReceiverProperties properties)
        {
            Assembly a = Assembly.GetExecutingAssembly();
            IDictionary<Type, Exception> exceptions = SPHealthAnalyzer.RegisterRules(a);
        }
        public override void FeatureDeactivating(SPFeatureReceiverProperties properties)
        {
            Assembly a = Assembly.GetExecutingAssembly();
            IDictionary<Type, Exception> exceptions = SPHealthAnalyzer.UnregisterRules(a);
        }
    }
} 

 

Rethinking the Intranet

When you look at intranet sites, what do you think? What makes for a good intranet experience? Typically, I hear things like “it has to have an intuitive navigation” or something to do with the aesthetics of the site. How well a site is organized is important, right? Sure, that makes sense.

Thinking about it now, I wonder, have sites really changed a lot in the last 10 years? There have been some new bells and whistles; new shiny objects that made them run faster, look prettier, animate them, or something like this. In the end though, intranet sites are generally the same. They typically have a deep hierarchy of information that seems to make sense to someone at some point, maybe organized by department or function. After launch, they seem to fall short of being that potential revelation envisioned when everyone looked at the mock-ups of the new pretty graphics and layout.

Let’s think about this in another way. Remember the VCR? For some reason, the clock on the VCR was notorious for users having difficulty setting it to the correct time. It seems silly today, but someone like my grandmother saw the VCR revolution come and go, and she never once learned how to set the correct time on it.

I see parallels with the VCR and intranet here. Engineers and usability experts spent years and countless revisions to their products and instruction manuals trying to make the process of setting the time on a VCR easy and intuitive to do. It’s a big deal, because if your VCR didn’t have the correct time, you couldn’t take advantage of features like the setting the timer to record programs. If you weren’t taking advantage of the advanced features, you might not buy a newer VCR later when one is released with more exciting new features.

The result was engineers and usability experts spending countless hours redesigning the product and writing better instructions for these users, adding better prompts and wizards on the screen hoping to guide users to set the VCR’s time. No matter what they did though, every time I visited my grandmother, her VCR continued to flash “12:00″, and I would have to set the time for her.

Eventually they finally had a thought, which must have been something like, “let’s see if we can find a way to have the VCR figure out what time it is and just set the clock automatically without bothering the user at all.” Now they finally uncovered a usable state. The result: this became the standard for all VCRs and future devices like BlueRays and PVRs! (When was the last time you had to set the time on something besides maybe your car radio?)

It seems that we are doing for intranets today like they used to do for VCRs. Think of some of the motivators for an intranet project: usually it has something to do with the intranet being in some state of a mess, content is not organized, people can’t find information, and no one really uses it. Then we deploy some usability experts and information architects to plan the intranet correctly this time. Thus, the project is justified and everyone dreams of all the wonderful things a new intranet will do for their company. I’ve even heard ludicrous statistics like “information workers spend up to X hours each week searching for information” justifying how bad things are and what value they assume a new intranet would deliver. I am typically suspicious of these numbers and studies, and more so when they are just incorporated in to a project vision.

To me, it feels like we send in these information architects and usability experts to do what the VCR people did years ago: trying to tweak a challenging structure to make users adapt to it. Think about it like this: after all these years of redoing intranets and spending so much time on organizing content, wouldn’t we as an industry have figured it out by now if this were the right track? Sure, there are incremental improvements, but with the exponential rate content and information is being created and added to intranets, this just isn’t sustainable. The clock continues to flash “12:00″ for too many intranets.

It’s time for a major shift, to look at a new paradigm. What if an intranet could set its own time? Instead of making a user navigate some crazy menu or deep hierarchy to go to the information, what if we brought the information to them?

Picture an experience where a user, any user, has a single landing page with a couple focus areas. You design each focus area around some core focus that you abstracted enough so they are applicable to anyone in the company. Yet, each user has their own experience in these areas. The page surfaces information that is important or relevant to the user within their given context; in other words, connecting the right information to the right people, at just the right time. I call this whole concept and guiding principle knowledge discovery.

We need information architects and usability experts to drive this revolution, this paradigm shift, uncovering the personalization answers and building out a completely new experience built around knowledge discovery. We are just burning cycles having them try and figure out a new more effective way to teach users how to discover content using the same constructs.

Setting up Shortcuts to the SharePoint Root

One thing I notice SharePoint administrators and developers frequently do is place a bunch of shortcuts on the desktop or taskbar of a SharePoint server to point to places like the SharePoint root directory or install directory. To be honest, I always find these icons and shortcuts to clutter the desktop or taskbar, and therefore a little inefficient. Let’s face it: this technique is a little out dated and sloppy.

What I like to do on any SharePoint server I’m working on is add the shortcuts I need to that top left “link” area of Windows Explorer. Then the shortcuts are available in any window that deals with files, from any program. For me, it’s very efficient. With a little bit of PowerShell, it’s quick and easy to set up:

$wshell = New-Object -comObject WScript.Shell
$link = $wshell.CreateShortcut($env:USERPROFILE + “\Links” + “\SharePoint Root.lnk”)
$link.Description = ‘SharePoint 2010 Root Directory (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14)’
$link.TargetPath = “C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14″
$link.Save()

Managing SharePoint 2010 Managed Metadata

A centralized enterprise-wide taxonomy is one of the great additions to SharePoint in 2010 known as the managed metadata service application. This capability provides a consistent way to classify content, sites, and other data across sites and farms. It also allows for developers to build applications that consume metadata services to make an organization’s data more manageable or introduce social computing capabilities.

The product exposes several ways to delegate management of the taxonomy or a branch of the taxonomy – referred to in SharePoint as a term set – and this is great for metadata structures that departments or business units manage. I’ve found the product to come up short though with configuration or release management of a managed metadata term store. This is particularly useful as part of a code promotion process. As developers build solutions that consume the metadata service to expose or interact with metadata, they reference a GUID for a term in the metadata store. SharePoint doesn’t export term sets and the Excel import sample doesn’t include a GUID column, so it is challenging for a developer to isolate their solution for testing and deployment.

For term sets I want to include in release management, I created a feature that will populate and update terms, term sets, and groups declared in XML. To avoid a lot of error checking in my code I created a schema to validate the XML against. You can get a copy of my schema here. The following is a sample of the XML I use to declare metadata as part of my feature that populates the managed metadata service application with it.

<?xml version=“1.0” encoding=“utf-8“ ?>
<ManagedMetadata>
  <Group Name=“Test Group 1” Lcid=“1033” Description=“Test Description 1“>
  <TermSet Name=“Test Term Set 1” Description=“Test Description 1” Guid=“431A8088-E43D-4D45-8B84-E0885C84EBE9” Lcid=“1033“>
   <Term Name=“Test Term 1” Description=“Test Description 1” Guid=“A661F7B1-D860-4EDA-884E-3900AEDFF7C4” Lcid=“1033“>
<Term Name=“Test Term 1A” Guid=“57DEA1DA-ACB6-4C6A-8F3E-C4002BDB0F46” Lcid=“1033“>
       <Label Name=“Test Label” Lcid=“1033“ />
      </Term>
    </Term>
   </TermSet>
</Group>
</ManagedMetadata>

To parse the XML and populate or update the managed metadata store, I create a utility class to handle feature activation and update events. I added a property to the feature manifest with the key “MetadataServiceAppKey” to specify the managed metadata service application name. I also supported multilingual capabilities, as my current customer supports several languages.

The code is pretty self-explanatory (I hope!), so I’ll just give a quick overview here. For each group, term set, term, and label the code checks if one already exists. If not, it creates a group, term set, term, or label. If one does exist, the code checks if the name or description has changed, if the term was depreciated, or if the term or term set was moved. It then updates as appropriate. I chose not to delete terms through the code and decided that adding the deprecating attribute should suffice. Below is the code in my utility class.

private const string LabelElementName = “Label”;
private const string TermElementName = “Term”;
private const string MetadataServiceAppKey = “MetadataServiceAppKey”;
private const string DeprecateKey = “Deprecate”;
private const string DescriptionKey = “Description”;
private const string GuidKey = “Guid”;
private const string LcidKey = “Lcid”;
private const string NameKey = “Name”;
private string ManagedMetadataServiceName;
private TermStore MetadataTermStore;
private XmlNode RootNode;

public void ProcessTaxonomy(SPFeatureReceiverProperties properties)
{
  ManagedMetadataServiceName = properties.Feature.Properties[MetadataServiceAppKey].Value;
 Stream xmlStream = properties.Feature.Definition.GetFile(“Taxonomy\\Taxonomy.xml”);
  Stream xsdStream = properties.Definition.GetFile(“Taxonomy\\Taxonomy.xsd”);
  XmlDocument xmlDoc = new XmlDocument();
  XmlSchema xmlSchema = XmlSchema.Read(xsdStream, null);
  xmlDoc.Schemas.Add(xmlSchema);
  xmlDoc.Schemas.Compile();
  xmlDoc.Load(xmlStream);
  RootNode = xmlDoc.FirstChild.NextSibling;
  SPSite site = properties.Feature.Parent as SPSite;
  TaxonomySession session = new TaxonomySession(site);
  MetadataTermStore = session.TermStores[ManagedMetadataServiceName];
  xmlDoc.Validate(new ValidationEventHandler(XmlDocumentValidationHandler));
ProcessXml();
}

private void CreateTermSets(Group group, XmlNodeList termSetNodes)
{
  foreach (XmlNode termSetNode in termSetNodes)
  {
    int lcid = int.Parse(termSetNode.Attributes[LcidKey].Value);
    Guid guid = new Guid(termSetNode.Attributes[GuidKey].Value);
    TermSet termSet = MetadataTermStore.GetTermSet(guid);

    if (termSet != null)
    {
       if (!termSet.Name.Equals(termSetNode.Attribute[NameKey].Value))
         termSet.Name = termSetNode.Attributes[NameKey].Value;
       if (!termSet.Group.Id.Equals(group.Id))
         termSet.Move(group);
    }
    else
       termSet = group.CreateTermSet(
         termSetNode.Attributes[NameKey].Value,
         guid,
         lcid);
  if (termSetNode.Attributes[DescriptionKey] != null &&
    !string.IsNullOrEmpty(termSetNode.Attributes[DescriptionKey].Value) &&
    termSet.Description != termSetNode.Attributes[DescriptionKey].Value)
    termSet.Description = termSetNode.Attributes[DescriptionKey].Value;

  CreateTerms(termSet, termSetNode.ChildNodes);
}
}

private void CreateTerms(TermSetItem termSetItem, XmlNodeList termChildNodes)
{
  List<string> labels = new List<string>();
  Term containingTerm = termSetItem as Term;
  if (containingTerm != null)
  {
    foreach (Label label in containingTerm.Labels)
labels.Add(label.Value);
  }
foreach (XmlNode node in termChildNodes)
 {
   int lcid = int.Parse(node.Attributes[LcidKey].Value);
   if (node.NodeType == XmlNodeType.Element &&
node.Name.Equals(TermElementName))
{
     Guid guid = new Guid(node.Attributes[GuidKey].Value);
     Term term = MetadataTermStore.GetTerm(guid);
     if (term != null)
{
       if (!term.Name.Equals(node.Attributes[NameKey].Value))
term.Name = node.Attributes[NameKey].Value;
        if (!term.TermSet.Id.Equals(termSetItem.Id))
        {
          if (termSetItem is TermSet)
            term.Move(termSetItem as TermSet);
          else
            term.Move(termSetItem as Term);
        }
      }
      else
      {
        term = termSetItem.CreateTerm(
          node.Attributes[NameKey].Value,
lcid,
          guid);
      }
      if (node.Attributes[DescriptionKey] != null &&
        !string.IsNullOrEmpty(node.Attributes[DescriptionKey].Value) &&
        term.GetDescription(lcid) != node.Attributes[DescriptionKey].Value)
term.SetDescription(node.Attributes[DescriptionKey].Value, lcid);
      if (node.Attributes[DeprecateKey] != null)
      {
        bool deprecate = bool.Parse(node.Attributes[DeprecateKey].Value);
        if (deprecate != term.IsDeprecated)
          term.Deprecate(deprecate);
      }

      CreateTerms(term, node.ChildNodes);
}
    else
if (node.NodeType == XmlNodeType.Element &&
        node.Name.Equals(LabelElementName))
      {
        Label label;
        if (labels.Contains(node.Attributes[NameKey].Value))
        {
          label = containingTerm.Labels[node.Attributes[NameKey].Value];
          if (!label.Language.Equals(lcid))
            label = containingTerm.CreateLabel(node.Attributes[NameKey].Value, lcid, false);
        }
        else
        {
          label = containingTerm.CreateLabel(node.Attributes[NameKey].Value, lcid, false);
        }
    }
}
}

protected void XmlDocumentValidationHandler(object sender, ValidationEventArgs args)
{
  if (args.Severity == XmlSeverityType.Warning)
    throw new XmlSchemaValidationException(args.Message);
  if (args.Severity == XmlSeverityType.Error)
    throw new XmlSchemaValidationException(args.Message);
}

protected void ProcessXml()
{
  List<string> groups = new List<string>(MetadataTermStore.Groups.Count);
  foreach (Group group in MetadataTermStore.Groups)
    groups.Add(group.Name);
  foreach (XmlNode groupNode in RootNode.ChildNodes)
  {
    if (groupNode.NodeType == XmlNodeType.Element)
    {
      Group group;
      if (groups.Contains(groupNode.Attributes[NameKey].Value))
        group = MetadataTermStore.Groups[groupNode.Attributes[NameKey].Value];
      else
        group = MetadataTermStore.CreateGroup(
          groupNode.Attributes[NameKey].Value);
      if (groupNode.Attributes[DescriptionKey] != null &&
         !string.IsNullOrEmpty(groupNode.Attributes[DescriptionKey].Value) &&
        group.Description != groupNode.Attributes[DescriptionKey].Value)
        group.Description = groupNode.Attributes[DescriptionKey].Value;

      CreateTermSets(group, groupNode.ChildNodes);
      MetadataTermStore.CommitAll();
    }
  }
  MetadataTermStore.CommitAll();
}

Configuring FAST for SharePoint 2010 Using PowerShell

FAST for SharePoint 2010 enhances the regular SharePoint search by adding some cool stuff like document previews in the search results and item counts displayed in the refiners. Unlike the built in SharePoint search though, it is a little funky to configure and get up and running. Well, so it seemed based on the beta documentation I was working from.

Fortunately, I’ve scripted the configuration and simplified it! That wasn’t my original driver though. To be honest, it was the crummy database names it generated when I set it up through Central Admin. SharePoint taking the liberty with some database naming conventions is one of my biggest pet peeves with the product. It should always seek or at least provide the ability for user input to override its poor defaults, no matter which method they are using to configure it. The few cases it does not are a fail in my books.

Setup and Initial Configurations

My initial setup is on a farm using a SharePoint Server, FAST Admin Server, and SQL Server. On the SharePoint server, I installed the SharePoint 2010 prerequisites, SharePoint Server enterprise, Office Web Components, SharePoint Foundation and Server language packs, and then ran PSCONFIG.

On the FAST Admin Server, I installed the FAST prerequisites and FAST for SharePoint 2010. I then prepared the following XML deployment file and put it in the C:\FASTSearch\installer\scripts folder:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<deployment version=”14″ modifiedBy=”domain\farm-account” modifiedTime=”2010-05-25T13:10:00+01:00″ comment=”No Comment” xmlns=”http://www.microsoft.com/enterprisesearch” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://www.microsoft.com/enterprisesearch/deployment.xsd”>
<instanceid>FASTSearchTestFarm</instanceid>
<connector-databaseconnectionstring>
<![CDATA[jdbc:sqlserver://sql-server:1433; DatabaseName=SPTEST_FASTAdmin]]>
</connector-databaseconnectionstring>
<host name=”fast-server.domain.com”>
<admin />
<document-processor processes=”4″/>
<content-distributor id=”0″ />
<indexing-dispatcher />
<crawler role=”single” />
<webanalyzer server=”true” max-targets=”1″ link-processing=”true” lookup-db=”true” redundant-lookup=”false” />
<searchengine row=”0″ column=”0″ />
<query />
</host>
<searchcluster>
<row id=”0″ index=”primary” search=”true” />
</searchcluster>
</deployment>

With the deployment configuration file ready, I opened a FAST PowerShell command and ran as administrator, executing the following PSConfig found in the C:\FASTSearch\installer\scripts directory to configure the FAST admin server:

.\psconfig.ps1 -action i -roleName admin -userName domain\farm-account -localMachineName fast-server.domain.com -databaseConnectionString sql-server -databaseName SPTEST_FASTAdmin -deploymentFile C:\FASTSearch\installer\scripts\deployment.xml

The PowerShell prompts for a self-signed certificate password, and in this case I entered “Pa$$w0rd” for simplicity of the blog post. Take note of the C:\FASTSearch\Install_Info.txt for the farms service references.

A couple of files are needed to be copied from the FAST server to the SharePoint server. Copy the connector PowerShell script from C: \FASTSearch\installer\scripts\securefastsearchconnector.ps1 and the FAST certificate from C:\FASTSearch\data\data_security\cert\FASTSearchCert.pfx. I copied both files to the C:\FASTSearch directory I created on my SharePoint server. Finally, for good measure, restart the FAST server to complete the initial configuration.

SharePoint Service Application Configuration

On the SharePoint server, open the PowerShell Integrated Scripting Environment (Windows PowerShell ISE), running as administrator, and load the SharePoint cmdlets. Tip: a quick way to load all the SharePoint cmdlets in the ISE is to open and run this PowerShell script: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\POWERSHELL\Registration\SharePoint.ps1

My script is a little crude and rough still, and I’m sure you can improve it some for your environment. But hopefully it proves to be a good start for you! In the ISE, paste, edit, and run the following script that will create the query and connector service applications.

$FASTDirectory = “C:\FASTSearch”
$ServiceAppPoolAccount = “domain\farm-account”
$AdminServiceAppPoolAccount = $ServiceAppPoolAccount

#FAST Configuration Details
$ContentDistributorUri = “fast-server.domain.com:13391″
$AdminServiceUri = “http://fast-server.domain.com:13257″

$QueryServiceUri = “http://fast-server.domain.com:13287″
$ResourceStoreUri = “http://fast-server.domain.com:13255″
$AdminServiceAuthUser = $ServiceAppPoolAccount

#Databases
$DatabaseServer = “sql-server”
$DatabasePrefix = “SPTEST_”
$UsageApplicationDatabaseName = $DatabasePrefix + “UsageApplication”
$StateServiceApplicationDatabaseName = $DatabasePrefix + “StateServices”
$ConnectorServiceDatabaseName = $DatabasePrefix + “FASTConnectorService”
$QueryServiceDatabaseName = $DatabasePrefix + “FASTQueryService”

$ConnectorServiceApplicationName = “FAST Connector Service”
$QueryServiceApplicationName = “FAST Query Service”
$StateServiceApplicationName = “State Service”
$UsageApplicatoinName = “SharePoint Usage Application”
$ConnectorCollection = “sp”

#Create Usage Application
Write-Host “Creating usage application”
New-SPUsageApplication -Name $UsageApplicatoinName -DatabaseServer $DatabaseServer -DatabaseName $UsageApplicationDatabaseName

#Create State Service Application
Write-Host “Creating state service application”
$StateServiceDB = New-SPStateServiceDatabase -Name $StateServiceApplicationDatabaseName
$StateServiceApp = New-SPStateServiceApplication -Name $StateServiceApplicationName -Database $StateServiceDB
New-SPStateServiceApplicationProxy -Name $StateServiceApplicationName -ServiceApplication $StateServiceApp -DefaultProxyGroup

#Create App Pools
Write-Host “Creating app pools”
$ServiceAppPool = New-SPServiceApplicationPool -Name “ServiceAppPool” -Account $ServiceAppPoolAccount
$AdminServiceAppPool = New-SPServiceApplicationPool -Name “AdminServiceAppPool” -Account $AdminServiceAppPoolAccount

#Create Query Search Application
Write-Host “Creating query search service application”
$SearchQueryServiceApp = New-SPEnterpriseSearchServiceApplication -Name $QueryServiceApplicationName -ApplicationPool $ServiceAppPool -AdminApplicationPool $AdminServiceAppPool -DatabaseName $QueryServiceDatabaseName -DatabaseServer $DatabaseServer
New-SPEnterpriseSearchServiceApplicationProxy -Name “$QueryServiceApplicationName Proxy” -Uri $SearchQueryServiceApp.Uri.AbsoluteUri

#Create Connector Search Application
Write-Host “Creating connector search service application”
$SearchConnectorServiceApp = New-SPEnterpriseSearchServiceApplication -Name $ConnectorServiceApplicationName -ApplicationPool $ServiceAppPool -AdminApplicationPool $AdminServiceAppPool -DatabaseName $ConnectorServiceDatabaseName -DatabaseServer $DatabaseServer -SearchApplicationtype “ExtendedConnector”
New-SPEnterpriseSearchServiceApplicationProxy -Name “$ConnectorServiceApplicationName Proxy” -Uri $SearchConnectorServiceApp.Uri.AbsoluteUri

Write-Host “Starting search service instance”
$SearchServiceInstance = Get-SPEnterpriseSearchServiceInstance –local
Start-SPEnterpriseSearchServiceInstance -Identity $SearchServiceInstance
Set-SPEnterpriseSearchAdministrationComponent -SearchApplication $SearchConnectorServiceApp -SearchServiceInstance $SearchServiceInstance
Set-SPEnterpriseSearchServiceApplication -Identity $ConnectorServiceApplicationName -DefaultSearchProvider “FASTSearch”

$SearchConnectorServiceApp | Get-SPEnterpriseSearchAdministrationComponent | Set-SPEnterpriseSearchAdministrationComponent -SearchServiceInstance $SearchServiceInstance
New-SPEnterpriseSearchExtendedConnectorProperty -Name “ContentDistributor” -SearchApplication $SearchConnectorServiceApp -value $ContentDistributorUri
New-SPEnterpriseSearchExtendedConnectorProperty -Name “Collection” -SearchApplication $SearchConnectorServiceApp -value $ConnectorCollection

Write-Host “Setting connector search crawl topology”
$CrawlDatabase0 = ([array]($SearchConnectorServiceApp | Get-SPEnterpriseSearchCrawlDatabase))[0]
$InitialCrawlTopology = $SearchConnectorServiceApp | Get-SPEnterpriseSearchCrawlTopology
$CrawlTopology = $SearchConnectorServiceApp | New-SPEnterpriseSearchCrawlTopology
$CrawlTopology | New-SPEnterpriseSearchCrawlComponent -CrawlDatabase $CrawlDatabase0 -SearchServiceInstance $SearchServiceInstance -IndexLocation $SearchServiceInstance.DefaultIndexLocation
Write-Host “Setting the new crawl topology to active”
$CrawlTopology | Set-SPEnterpriseSearchCrawlTopology –Active
Write-Host “Waiting for the new crawl topology to initialize”
While($InitialCrawlTopology.State -ne “Inactive”){}
$InitialCrawlTopology.Delete()

#Configure Query Search Application
$SearchQueryServiceApp | Get-SPEnterpriseSearchAdministrationComponent | Set-SPEnterpriseSearchAdministrationComponent -SearchServiceInstance $SearchServiceInstance
Set-SPEnterpriseSearchServiceApplication -identity $QueryServiceApplicationName -defaultsearchprovider “FASTSearch”
Set-SPEnterpriseSearchExtendedQueryProperty -SearchApplication $QueryServiceApplicationName -Identity “FASTSearchAdminServiceLocation” -Value $AdminServiceUri
Set-SPEnterpriseSearchExtendedQueryProperty -SearchApplication $QueryServiceApplicationName -Identity “FASTSearchQueryServiceLocation” -Value $QueryServiceUri
Set-SPEnterpriseSearchExtendedQueryProperty -SearchApplication $QueryServiceApplicationName -Identity “FASTSearchResourceStoreLocation” -Value $ResourceStoreUri
Set-SPEnterpriseSearchExtendedQueryProperty -SearchApplication $QueryServiceApplicationName -Identity “FASTSearchAdminServiceAuthenticationUser” -Value $AdminServiceAuthUser
Write-Host “Setting query search crawl topology”
$InitialCrawlTopology = $SearchQueryServiceApp | Get-SPEnterpriseSearchCrawlTopology –Active
$CrawlDatabase0 = ([array]($SearchQueryServiceApp | Get-SPEnterpriseSearchCrawlDatabase))[0]
$CrawlTopology = $SearchQueryServiceApp | New-SPEnterpriseSearchCrawlTopology
$CrawlComponent = New-SPEnterpriseSearchCrawlComponent -CrawlTopology $CrawlTopology -CrawlDatabase $CrawlDatabase0 -SearchServiceInstance $SearchServiceInstance -IndexLocation $SearchServiceInstance.DefaultIndexLocation
Write-Host “Setting the new crawl topology to active”
$CrawlTopology | Set-SPEnterpriseSearchCrawlTopology –Active
Write-Host “Waiting for the new crawl topology to initialize”
While($InitialCrawlTopology.State -ne “Inactive”){}
$InitialCrawlTopology.Delete()

Write-Host “Setting the query topology”
$QueryTopology = New-SPEnterpriseSearchQueryTopology -SearchApplication $SearchQueryServiceApp -Partitions 1
$IndexPartition = Get-SPEnterpriseSearchIndexPartition -QueryTopology $QueryTopology
$QueryComponent = New-SPEnterpriseSearchQueryComponent -QueryTopology $QueryTopology -IndexPartition $IndexPartition -SearchServiceInstance $SearchServiceInstance -IndexLocation $SearchServiceInstance.DefaultIndexLocation
$PropertyDatabase0 = ([array]($SearchQueryServiceApp | Get-SPEnterpriseSearchPropertyDatabase))[0]
$IndexPartition | Set-SPEnterpriseSearchIndexPartition -PropertyDatabase $PropertyDatabase0
$QueryTopology.Activate()

#Start the Search Query and Site Settings Service
Get-SPEnterpriseSearchQueryAndSiteSettingsServiceInstance -Local | Start-SPEnterpriseSearchQueryAndSiteSettingsServiceInstance

#Export SharePoint Cert & Import FAST Cert
Write-Host “Exporting the SharePoint Certificate”
$SharePointCertFileName = Join-Path -Path $FASTDirectory -ChildPath “SharePointCert.cer”
$SharePointCert = (Get-SPSecurityTokenServiceConfig).LocalLoginProvider.SigningCertificate
$SharePointCert.Export(“cert”) | Set-Content -encoding byte $SharePointCertFileName
cd\
cd $FASTDirectory
. .\securefastsearchconnector.ps1 –certPath “.\FASTSearchCert.pfx” –ssaName $ConnectorServiceApplicationName –username $AdminServiceAuthUser

After this script finishes executing, copy the SharePoint certificate from c:\FASTSearch\SharePointCert.cer on the SharePoint server to the FAST server in c:\FASTSearch\installer\scripts\SharePointCert.cer

Enable Click-Through Relevancy and Import SharePoint Certificate on FAST

On the FAST server, open the PowerShell ISE running as administrator and paste, edit, and execute the following PowerShell:

$SharePointIdentity = “domain\farm-account”
$ScriptsDirectory = “C:\FASTSearch\installer\scripts”
$SharePointCertPath = $ScriptsDirectory | Join-Path -ChildPath “SharePointCert.cer”

cd\
cd $ScriptsDirectory
.\ConfigureSharePointAuthorization.ps1 -InstalledMode Advanced -SharePointUserIdentity $SharePointIdentity

$trustedPeopleCertStore = New-Object System.Security.Cryptography.X509Certificates.X509Store(“TrustedPeople”, [System.Security.Cryptography.X509Certificates.StoreLocation]::LocalMachine)
$trustedPeopleCertStore.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadWrite)
$SharePointCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$SharePointCert.Import($SharePointCertPath)
$trustedPeopleCertStore.Add($SharePointCert)
$trustedPeopleCertStore.Close()

In Central Admin, double check that the query service application is associated with the default service application group or the web application the FAST search center is hosted in. You need to create service applications for the Office Web Components if you want the view previews of your Office documents in the search results.

You can add content sources for SharePoint and other content in the connector service application and crawl some content to begin testing search. Don’t forget to add the people profile content source (i.e. sps3://mysite) to the query service application instead of the connector service application for people searches. From there, you’re all set!