Remove old data form the index for Sitecore 6.2 (Update-4) and 6.3

Previously I published an article explaining how to fix a glitch in Sitecore index that keeps old data after updating an item. In this article I updated the code since some issues were fixed in Sitecore 6.2 (Update-4) and 6.3.

Starting from 6.2 (Update-4) and 6.3 the _template field in Lucene index stores an item ID in ShortID format. Now it’s even easier to customize the DatabaseCrawler as the following methods were made virtual: AddItem, IndexVersion, DeleteItem and DeleteVersion. Also fields _hasIncludes, _hasExludes and _templateFilder were made protected which helps to shrink the fix even more.

Here is the code that fixes the outlined problem for these releases:

Code Snippet

1. namespace Lucene.Search.Crawlers
2. {
3.     public class DatabaseCrawler : Sitecore.Search.Crawlers.DatabaseCrawler
4.     {
5.         protected override void AddMatchCriteria(Net.Search.BooleanQuery query)
6.         {
7.             query.Add(new TermQuery(new Term(BuiltinFields.Database, RootItem.Database.Name)), BooleanClause.Occur.MUST);
8.             query.Add(new TermQuery(new Term(BuiltinFields.Path, ShortID.Encode(RootItem.ID).ToLowerInvariant())), BooleanClause.Occur.MUST);
9.             if (this._hasIncludes || this._hasExcludes)
10.             {
11.                 foreach (KeyValuePair<string, bool> pair in this._templateFilter)
12.                 {
13.                     query.Add(new TermQuery(new Term(BuiltinFields.Template, ShortID.Encode(pair.Key).ToLowerInvariant())), pair.Value ? BooleanClause.Occur.SHOULD : BooleanClause.Occur.MUST_NOT);
14.                 }
15.             }
16.         }
17.  
18.         protected Item RootItem
19.         {
20.             get
21.             {
22.                 return Sitecore.Data.Managers.ItemManager.GetItem(Root, Sitecore.Globalization.Language.Invariant,
23.                                                                   Sitecore.Data.Version.Latest,
24.                                                                   Sitecore.Data.Database.GetDatabase(Database),
25.                                                                   Sitecore.SecurityModel.SecurityCheck.Disable);
26.             }
27.         }
28.  
29.         protected override Query GetVersionQuery(ID id, string language, string version)
30.         {
31.             Assert.ArgumentNotNull(id, "id");
32.             Assert.ArgumentNotNullOrEmpty(language, "language");
33.             Assert.ArgumentNotNullOrEmpty(version, "version");
34.             BooleanQuery query = new BooleanQuery();
35.             query.Add(new TermQuery(new Term(BuiltinFields.ID, GetItemID(id, language, version).ToLowerInvariant())), BooleanClause.Occur.MUST);
36.             this.AddMatchCriteria(query);
37.             return query;
38.         }
39.     }
40. }

Basically you need to make sure that the result of GetItemID method is in lower case as well as a term for _path field, in AddMatchCriteria method, is constructed with lower case query.

Adding custom fields to the index

In this post I want to show how to address a missing feature that was a part of “old” lucene index implementation. This article will provide an example how one can customize Lucene search configuration so that it’s possible to add custom fields to the index.

First off, let’s create a configuration that would allow us to add additional fields to the indexed data.

<index id="News" type="Sitecore.Search.Index, Sitecore.Kernel">
  <param desc="name">$(id)</param>
  <param desc="folder">_news</param>
  <Analyzer ref="search/analyzer" />
  <locations hint="list:AddCrawler">
    <examples-news type="LuceneExamples.DatabaseCrawler,LuceneExamples">
      <Database>web</Database>
      <Root>/sitecore/content</Root>
      <IndexAllFields>true</IndexAllFields>
      <include hint="list:IncludeTemplate">
        <news>{788EF1BE-B71E-4D59-9276-50519BD4F641}</news>
        <tag>{4DD970FB-2695-4E50-96F3-A766F7D6CAF1}</tag>
      </include>
      <fields hint="raw:AddCustomField">
        <field luceneName="author" storageType="no" indexType="tokenized">__updated by</field>
        <field luceneName="changed" storageType="yes" indexType="untokenized">__updated</field>
      </fields>
    </examples-news>
  </locations>
</index>

There is a new configuration section in this example. It’s <fields> section that introduces two fields “author” and “changed”. These fields will be added to a fields collection of each indexed item. Basically, there is AddCustomField method that gets called for every <field> configuration entry to identify a custom field that is going to be added to the fields collection.

Description of configuration attributes:

• luceneName  is a field name that appears in lucene index.
• storageType  is a storage type for lucene field. It can have the following values:
  • no
  • yes
  • compress
• indexType  is an index type for lucene field. It can have the following values:
  • no
  • tokenized
  • untokenized
  • nonorms

Refere to Lucene documentation to find out what each of these options mean: store and index.

Now all you need to do is to loop through the collection of custom fields in the overridden AddAllFields method and add them to the indexed data.

I created a custom class called CustomField that helps to manage custom field entries. Below is the example of this class as well as additional methods for extended DatabaseCrawler. Since code for the DatabaseCrawler was already published in this blog post, I’m not going to duplicate it here.

Here is a code for CustomField class.

using System.Xml;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Xml;
using Lucene.Net.Documents;

namespace LuceneExamples
{
   public class CustomField
   {
      public CustomField()
      {
         FieldID = ID.Null;
         FieldName = "";
         LuceneFieldName = "";
      }

      public ID FieldID
      {
         get;
         private set;
      }

      public string FieldName { get; private set; }

      public Field.Store StorageType { get; set; }

      public Field.Index IndexType { get; set; }

      public string LuceneFieldName { get; private set; }

      public static CustomField ParseConfigNode(XmlNode node)
      {
         CustomField field = new CustomField();
         string fieldName = XmlUtil.GetValue(node);
         if (ID.IsID(fieldName))
         {
            field.FieldID = ID.Parse(fieldName);
         }
         else
         {
            field.FieldName = fieldName;
         }
         field.LuceneFieldName = XmlUtil.GetAttribute("luceneName", node);
         field.StorageType = GetStorageType(node);
         field.IndexType = GetIndexType(node);

         if (!IsValidField(field))
         {
            return null;
         }

         return field;
      }

      public string GetFieldValue(Item item)
      {
         if (!ID.IsNullOrEmpty(FieldID))
         {
            return item[ID.Parse(FieldID)];
         }
         if(!string.IsNullOrEmpty(FieldName))
         {
            return item[FieldName];
         }
         return string.Empty;
      }

      private static bool IsValidField(CustomField field)
      {
         if ((!string.IsNullOrEmpty(field.FieldName) || !ID.IsNullOrEmpty(field.FieldID)) && !string.IsNullOrEmpty(field.LuceneFieldName))
         {
            return true;
         }
         return false;
      }

      private static Field.Index GetIndexType(XmlNode node)
      {
         string indexType = XmlUtil.GetAttribute("indexType", node);
         if (!string.IsNullOrEmpty(indexType))
         {
            switch (indexType.ToLowerInvariant())
            {
               case "no":
                  return Field.Index.NO;
               case "tokenized":
                  return Field.Index.TOKENIZED;
               case "untokenized":
                  return Field.Index.UN_TOKENIZED;
               case "nonorms":
                  return Field.Index.NO_NORMS;
            }
         }
         return Field.Index.TOKENIZED;
      }

      private static Field.Store GetStorageType(XmlNode node)
      {
         string storage = XmlUtil.GetAttribute("storageType", node);
         if (!string.IsNullOrEmpty(storage))
         {
            switch (storage.ToLowerInvariant())
            {
               case "no":
                  return Field.Store.NO;
               case "yes":
                  return Field.Store.YES;
               case "compress":
                  return Field.Store.COMPRESS;
            }
         }
         return Field.Store.NO;
      }
   }
}

And the code for additional methods for DatabaseCrawler.

/// <summary>
/// Loops through the collection of custom fields and adds them to fields collection of each indexed item.
/// </summary>
/// <param name="document">Lucene document</param>
/// <param name="item">Sitecore data item</param>
private void AddCustomFields(Document document, Item item)
{
   foreach(CustomField field in _customFields)
   {
      document.Add(CreateField(field.LuceneFieldName, field.GetFieldValue(item), field.StorageType, field.IndexType, Boost));
   }
}

/// <summary>
/// Creates a Lucene field.
/// </summary>
/// <param name="fieldKey">Field name</param>
/// <param name="fieldValue">Field value</param>
/// <param name="storeType">Storage option</param>
/// <param name="indexType">Index type</param>
/// <param name="boost">Boosting parameter</param>
/// <returns></returns>
private Fieldable CreateField(string fieldKey, string fieldValue, Field.Store storeType, Field.Index indexType, float boost)
{
   Field field = new Field(fieldKey, fieldValue, storeType, indexType);
   field.SetBoost(boost);
   return field;
}

/// <summary>
/// Parses a configuration entry for a custom field and adds it to a collection of custom fields.
/// </summary>
/// <param name="node">Configuration entry</param>
public void AddCustomField(XmlNode node)
{
   CustomField field = CustomField.ParseConfigNode(node);
   if (field == null)
   {
      throw new InvalidOperationException("Could not parse custom field entry: " + node.OuterXml);
   }
   _customFields.Add(field);
}

Last thing that is left to do is to call AddCustomFields method from AddAllFields one.

protected override void AddAllFields (Documentdocument, Itemitem, bool versionSpecific)
{
    ………………………………………
    AddCustomFields(document, item);
}

You can take it even further and add support for some field interpreter for each field configuration entry.

Hope you'll find it useful.

Language filtered Multilist field

Recently I happened to help one of our clients to create a custom Multilist field that gives you selecting options only if item has  at least one translated version in a current content language. From content author’s point of view it makes a lot of sense. Why to give them an option that is not useful?

It’s being a while since I created a custom field that has to take into account content language selection. The main challenge for me was to retrieve that content language. I did remember that there is a property that the Content Editor (CE) sets at run-time for every field but could not recall its name. So after several minutes of browsing my code storage of all samples for all Sitecore versions, I finally found it. So, here are the properties that you need to define in your custom field if you are planning to use them afterwards:

- ItemLanauage – represents a content language selected in the CE.

- ItemID – contains the item ID the field belongs to.

- ItemVersion – contains selected item version.

- ReadOnly – indicates if field is a readonly. If it is, then it will be grayed out and not editable.

- Source – represents the source value from field definition on a data template.

- FieldID – contains the field ID.

In my example I needed only ItemLanguage property. When I put things together the code looked like this:

using Sitecore.Shell.Applications.ContentEditor;
using Sitecore.Data.Items;

namespace Sitecore.Shell.Applications.ContentEditor.CustomExtensions
{
    public class LanguageFilteredMultilist : Sitecore.Shell.Applications.ContentEditor.MultilistEx
    {
        #region Overrides

        protected override Item[] GetItems(Item current)
        {              
            Item[] items = base.GetItems(current);

            var filteredItems = items.Where(item => 
                    {
                        string lang = ItemLanguage;
                        if (!string.IsNullOrEmpty(lang))
                        {
                            var versions = Sitecore.Data.Managers.ItemManager.GetVersions(item, Sitecore.Data.Managers.LanguageManager.GetLanguage(lang, item.Database));
                            return versions.Count > 0;
                        }
                        return false;
                    }
                );   
            
            return filteredItems.ToArray<Item>();
        }

        #endregion Overrides

        // Content Editor sets this property. It has a content language from the Content Editor.
        public string ItemLanguage
        {
            get;
            set;
        }
    }
}

Simple enough isn’t it. Don’t forget to add your custom field to /App_Config/FieldTypes.config file if it’s supposed to contain references to other items. In my case it should be added since it’s a multilist. If you forget to do it, the LinkDatabase won’t update references for your field.

That’s all for now.

Teach User Manager how to search by email

A couple of days ago one interesting issue popped out in my Inbox. Our customer wanted to search users by email address in User Manager application. Quite legitimate request. Especially if you have an integration with AD or CRM or whatever external security system with large number of users. I tried a couple of configuration changes with no luck. Then checked our support portal knowledge base and saw several request like that. I was surprised by the fact that you cannot search users by email.

After spending several hours investigating how search functionality in User Manager works and doing some "shaman dancing" around it, I ended up with a solution that requires to override a standard Sitecore.Security.Accounts.UserProvider and change its method called "GetUsersByName". As it turned out, search functionality of User Manager calls the GetUserByName method when one searches for a user.

This is how it looks:

namespace Sitecore.SharedSource.Security.Accounts

{

public class UserProvider : Sitecore.Security.Accounts.UserProvider

{

protected override IEnumerable GetUsersByName(int pageIndex, int pageSize, string userNameToMatch)

{

return FindUsers(pageIndex, pageSize, userNameToMatch);

}

protected IEnumerable FindUsers(int pageIndex, int pageSize, string userNameToMatch)

{

Assert.ArgumentNotNull(userNameToMatch, "userNameToMatch");

IEnumerable users = null;

string userWithNoWildcard = StringUtil.RemovePostfix(Settings.Authentication.VirtualMembershipWildcard, StringUtil.RemovePrefix(Settings.Authentication.VirtualMembershipWildcard, userNameToMatch));

if (Regex.IsMatch(userWithNoWildcard, @"^([0-9a-zA-Z]([-.\w]*[0-9a-zA-Z])*@([0-9a-zA-Z][-\w]*[0-9a-zA-Z]\.)+[a-zA-Z]{2,9})$"))

{

users = findByEmail(pageIndex, pageSize, userWithNoWildcard);

}

else

{

users = findUsername(pageIndex, pageSize, userNameToMatch);

}

return users;

}

protected IEnumerable findUsername(int pageIndex, int pageSize, string username)

{

int total;

return new Enumerable(delegate

{

return Membership.FindUsersByName(username, pageIndex, pageSize, out total).GetEnumerator();

},

delegate(object user)

{

return User.FromName(((MembershipUser)user).UserName, false);

});

}

protected IEnumerable findByEmail(int pageIndex, int pageSize, string userToMatch)

{

int total;

return new Enumerable(delegate

{

return Membership.FindUsersByEmail(userToMatch, pageIndex, pageSize, out total).GetEnumerator();

},

delegate(object user)

{

return User.FromName(((MembershipUser)user).UserName, false);

});

}

}

}

That's it! Simple enough, isn't it.

Thanks to Alex Shyba who helped me with some useful code!

After compiling the code and moving a dll to the /bin folder, don't forget to change a reference in web.config file to point it to a new UserProvider class.


    <userManager defaultProvider="default" enabled="true">
      <providers>
        <clear />
        <add name="default">
          <x:attribute name="type">Sitecore.SharedSource.Security.Accounts.UserProvider, YOUR_DLL_HERE</x:attribute>
        </add>
      </providers>
    </userManager>

For those who consider this so easy so that it's now worth to create a VS project I've built a ready-to-go Sitecore package with a dll and include file inside. Just install it and give it a try ;).

This is a Sitecore package.

Enjoy!

Working with Lucene Search Index in Sitecore 6. Part III - Code examples

If you read my previous posts about Lucene search index, then you already know how to configure it and how it works in Sitecore application.

In this part we will take a look at API to see what can be achieved using the search index.

To search existing index we need to get an index instance somehow. I'm not going to show the code examples that you would write with old search index. If you're intrested in it, check out this article Lecene Search Engine.

Additional layer of API for new search index resides under Sitecore.Search namespace. In order to get a search index object, you would need to use members of SearchManager class.

To get an index by name use this line of code:

Index indx = Search.Manager.GetIndex(index_name);

If you want to use default system index, you can simply call SystemIndex property of SearchManager class. In order to use Sitecore API to look up for some info in the index, you need to created a search context.

It's easy to do by calling CreateSearchContext method of the index object we got previously. It's also possible to create a search context by using one for the constructors of IndexSearchContext class. In this case it will be easy to run search queries by passing a search index instance as a parameter to the search context.

To search information, we need to create a query and run it in the search context. Sitecore API has a few classes that you can use to build search queries. Let's take a look at each of them:

FullTextQuery - this type of query searches the index by "_content" field. All information from text fields (such as "Single-Line Text", "Rich Text", "Multi-Line Text", "text", "rich text", "html", "memo") is stored there. Data in this field are indexed and tokenized. Which means that the search operations running on these data are very efficient.

FieldQuery - this type of query allows you to search any field that was added to the index. By default database crawler adds all item fields to the index.

CombinedQuery - this type of query was designed to allow you create complex queries with additional conditions. For instance to find items which have specific work in title and belong to some category. When you add search queries to this type of query, you need to supply QueryOccurance parameter. It's Enum type that has the following members:

- Must - it's a logical AND operator in Lucene boolean query.

- MustNot - it's a logical NOT operator in Lucene boolean query.

- Should - it's a logical OR operation in Lucene boolean query.

You can read more about this operators in Query Parser Syntax article.

All of these query types are derived from QueryBase class.

There is one thing left until we jump from the theory to some code samples. To run defined queries, you need to use one of Search methods of IndexSearchContext object.

Now let's create a couple of samples to see a real code that goes behind the theory.

Sample 1: Searching text fields.

// Next samples will skip lines with getting the index instance and creating the search context.

// Get an index object

Index indx = SearchManager.GetIndex("my_index");

// Create a search context

using (IndexSearchContext searchContext = indx.CreateSearchContext())

{

// In following examples I will be using QueryBase class to create search queries.

FullTextQuery ftQuery = new FullTextQuery("welcome");

SearchHits results = searchContext.Search(ftQuery);

}

Sample 2: Searching item fields

Let's say we want to find items classified by some category. There is a trick searching by GUIDs so let's say our category is just a string name.

.....

// FieldQuery ctor accepts two parambers. First is a field name. The other one is a value we're looking for.

QueryBase query = FieldQuery("category", "slr");

SearchHits results = searchContext.Search(query);

.....

Sample 3: Searching by multiple conditions

Turned out that your category parameter is not enough to get required results. You client is screaming that there are too many items and business users cannot find ones they are looking for (is there anything they can find?).

Obviously you have some additional fields that can help to find more strict results. Let's say there is a rating field with values from 1 to 5.

That's where CombinedQuery gets into the game.

.....

// CombinedQuery object has Add method that should be used to add search queries to it. That's why we cannot use base class variable here.

CombinedQuery query = new CombinedQuery();

QueryBase catQuery = new FieldQuery("category", "slr");

QueryBase ratQuery = new FieldQuery("rating", "5");

query.Add(catQuery, QueryOccurance.Must);

query.Add(ratQuery, QueryOccurance.Must);

var hits = searchContext.Search(query);

.....

All results are presented as SearchHits object. Now you should use of following methods of SearchHits object to get the results as Sitecore items:

- FetchResults(int, int) - returns search results as SearchResultCollection. First parameter is a start position of an item you want to start fetching results from. Second one is count of items you want to fetch. By calling this function as mentioned below, you can get all results at once:

var results = hits.FetchResults(0, hits.Length);

- Slice(int) - returns all results as IEnumerable collection.

- Slice(int, int) - this method has similar signature to FetchResults but returns results as IEnumerable collection.

Here are a couple of examples the way you can transform SearchHits object into Sitecore items.

Sample 4: using FetchResults

.....

SearchResultCollection results = hits.FetchResults(0, hits.Length);

IEnumerable searchItems = from hit in results

select hit.GetObject();

}

.....

Sample 5: using Slice

.....

IList searchItems = List();

foreach(var hit in hits.Slice(0))

{

ItemUri itemUri = new ItemUri(hit.Url);

if (itemUri != null)

{

Item item = ItemManager.GetItem(itemUri.ItemID, itemUri.Language, itemUri.Version, Factory.GetDatabase(itemUri.DatabaseName));

if (item != null)

{

searchItems.Add(item);

}

}

}

.....

It's worth to mention that some variations of Search method of IndexSearchContext class can accept Lucene.Net.Search.Query as a search query parameter. It becomes very useful when you need to create a complex query which cannot be built with Sitecore query types.

Searching GUIDs.

New search index has lots of useful built-in fields that help to build strict queries.

Besides standard fields it has the following fields that contain GUIDs in ShortID format:

- _links - contains all references to current item

- _path - contains ShortIDs for every parent item in the path relative to current item

- _template - contains GUID of item's tempalte.

NOTE: this field is supposed to have ShortID value instead of GUID one. This field should not be used in combined queries prior to Sitecore 6.2 releases.

If you decide to add custom fields to your search index and they should have GUID values, you need to store them as ShortID in lower case format. Otherwise search will not be able to find any results. The reason why it happens is because Lucene recognizes GUIDs and applies special parsing for them. It works fine if search query has only one field to look into. If it's combined/complex query then it fails to find anything even if it's correct.

So, remember if you need to filter search results by template, you will have to customize DatabaseCrawler to add another field (e.g. _shorttemplateid) to store item template id in ShortID format.

Sample 1: Find all item references

.....

QueryBase query = FieldQuery("_links", ShortID.Encode(item.ID).ToLowerInvariant());

SearchHits results = searchContext.Search(query);

.....

Sample 6: Find all items based on specified template

.....

// Prior to Sitecore 6.2 release, you will need to add and use _shorttemplateid field

QueryBase query = FieldQuery("_template", ShortID.Encode(item.ID).ToLowerInvariant());

SearchHits results = searchContext.Search(query);

.....

Sample 7: Find items that are descendants of a specified one

.....

QueryBase query = FieldQuery("_path", ShortID.Encode(item.ID).ToLowerInvariant());

SearchHits results = searchContext.Search(query);

.....

Sample 8: Find items of a parent and belong to a specific template

.....

CombinedQuery query = new CombinedQuery();

query.Add(new FieldQuery("_shorttemplateid", ShortID.Encode(templateId).ToLowerInvariant()), QueryOccurance.Must);

query.Add(new FieldQuery("_path", ShortID.Encode(parent.ID).ToLowerInvariant())), QueryOccurance.Must);

.....

That's all I wanted to tell about Lucene search index in Sitecore 6. I hope it will help Lucene beginners to better understand the concept and get up to speed with Lucene search index abilities.

Enjoy!