Working with Lucene Search Index in Sitecore 6. Part III - Code examples

If you read my previous posts about Lucene search index, then you already know how to configure it and how it works in Sitecore application.

In this part we will take a look at API to see what can be achieved using the search index.

To search existing index we need to get an index instance somehow. I'm not going to show the code examples that you would write with old search index. If you're intrested in it, check out this article Lecene Search Engine.

Additional layer of API for new search index resides under Sitecore.Search namespace. In order to get a search index object, you would need to use members of SearchManager class.

To get an index by name use this line of code:

Index indx = Search.Manager.GetIndex(index_name);

If you want to use default system index, you can simply call SystemIndex property of SearchManager class. In order to use Sitecore API to look up for some info in the index, you need to created a search context.

It's easy to do by calling CreateSearchContext method of the index object we got previously. It's also possible to create a search context by using one for the constructors of IndexSearchContext class. In this case it will be easy to run search queries by passing a search index instance as a parameter to the search context.

To search information, we need to create a query and run it in the search context. Sitecore API has a few classes that you can use to build search queries. Let's take a look at each of them:

FullTextQuery - this type of query searches the index by "_content" field. All information from text fields (such as "Single-Line Text", "Rich Text", "Multi-Line Text", "text", "rich text", "html", "memo") is stored there. Data in this field are indexed and tokenized. Which means that the search operations running on these data are very efficient.

FieldQuery - this type of query allows you to search any field that was added to the index. By default database crawler adds all item fields to the index.

CombinedQuery - this type of query was designed to allow you create complex queries with additional conditions. For instance to find items which have specific work in title and belong to some category. When you add search queries to this type of query, you need to supply QueryOccurance parameter. It's Enum type that has the following members:

- Must - it's a logical AND operator in Lucene boolean query.

- MustNot - it's a logical NOT operator in Lucene boolean query.

- Should - it's a logical OR operation in Lucene boolean query.

You can read more about this operators in Query Parser Syntax article.

All of these query types are derived from QueryBase class.

There is one thing left until we jump from the theory to some code samples. To run defined queries, you need to use one of Search methods of IndexSearchContext object.

Now let's create a couple of samples to see a real code that goes behind the theory.

Sample 1: Searching text fields.

// Next samples will skip lines with getting the index instance and creating the search context.

// Get an index object

Index indx = SearchManager.GetIndex("my_index");

// Create a search context

using (IndexSearchContext searchContext = indx.CreateSearchContext())

{

// In following examples I will be using QueryBase class to create search queries.

FullTextQuery ftQuery = new FullTextQuery("welcome");

SearchHits results = searchContext.Search(ftQuery);

}

Sample 2: Searching item fields

Let's say we want to find items classified by some category. There is a trick searching by GUIDs so let's say our category is just a string name.

.....

// FieldQuery ctor accepts two parambers. First is a field name. The other one is a value we're looking for.

QueryBase query = FieldQuery("category", "slr");

SearchHits results = searchContext.Search(query);

.....

Sample 3: Searching by multiple conditions

Turned out that your category parameter is not enough to get required results. You client is screaming that there are too many items and business users cannot find ones they are looking for (is there anything they can find?).

Obviously you have some additional fields that can help to find more strict results. Let's say there is a rating field with values from 1 to 5.

That's where CombinedQuery gets into the game.

.....

// CombinedQuery object has Add method that should be used to add search queries to it. That's why we cannot use base class variable here.

CombinedQuery query = new CombinedQuery();

QueryBase catQuery = new FieldQuery("category", "slr");

QueryBase ratQuery = new FieldQuery("rating", "5");

query.Add(catQuery, QueryOccurance.Must);

query.Add(ratQuery, QueryOccurance.Must);

var hits = searchContext.Search(query);

.....

All results are presented as SearchHits object. Now you should use of following methods of SearchHits object to get the results as Sitecore items:

- FetchResults(int, int) - returns search results as SearchResultCollection. First parameter is a start position of an item you want to start fetching results from. Second one is count of items you want to fetch. By calling this function as mentioned below, you can get all results at once:

var results = hits.FetchResults(0, hits.Length);

- Slice(int) - returns all results as IEnumerable collection.

- Slice(int, int) - this method has similar signature to FetchResults but returns results as IEnumerable collection.

Here are a couple of examples the way you can transform SearchHits object into Sitecore items.

Sample 4: using FetchResults

.....

SearchResultCollection results = hits.FetchResults(0, hits.Length);

IEnumerable searchItems = from hit in results

select hit.GetObject();

}

.....

Sample 5: using Slice

.....

IList searchItems = List();

foreach(var hit in hits.Slice(0))

{

ItemUri itemUri = new ItemUri(hit.Url);

if (itemUri != null)

{

Item item = ItemManager.GetItem(itemUri.ItemID, itemUri.Language, itemUri.Version, Factory.GetDatabase(itemUri.DatabaseName));

if (item != null)

{

searchItems.Add(item);

}

}

}

.....

It's worth to mention that some variations of Search method of IndexSearchContext class can accept Lucene.Net.Search.Query as a search query parameter. It becomes very useful when you need to create a complex query which cannot be built with Sitecore query types.

Searching GUIDs.

New search index has lots of useful built-in fields that help to build strict queries.

Besides standard fields it has the following fields that contain GUIDs in ShortID format:

- _links - contains all references to current item

- _path - contains ShortIDs for every parent item in the path relative to current item

- _template - contains GUID of item's tempalte.

NOTE: this field is supposed to have ShortID value instead of GUID one. This field should not be used in combined queries prior to Sitecore 6.2 releases.

If you decide to add custom fields to your search index and they should have GUID values, you need to store them as ShortID in lower case format. Otherwise search will not be able to find any results. The reason why it happens is because Lucene recognizes GUIDs and applies special parsing for them. It works fine if search query has only one field to look into. If it's combined/complex query then it fails to find anything even if it's correct.

So, remember if you need to filter search results by template, you will have to customize DatabaseCrawler to add another field (e.g. _shorttemplateid) to store item template id in ShortID format.

Sample 1: Find all item references

.....

QueryBase query = FieldQuery("_links", ShortID.Encode(item.ID).ToLowerInvariant());

SearchHits results = searchContext.Search(query);

.....

Sample 6: Find all items based on specified template

.....

// Prior to Sitecore 6.2 release, you will need to add and use _shorttemplateid field

QueryBase query = FieldQuery("_template", ShortID.Encode(item.ID).ToLowerInvariant());

SearchHits results = searchContext.Search(query);

.....

Sample 7: Find items that are descendants of a specified one

.....

QueryBase query = FieldQuery("_path", ShortID.Encode(item.ID).ToLowerInvariant());

SearchHits results = searchContext.Search(query);

.....

Sample 8: Find items of a parent and belong to a specific template

.....

CombinedQuery query = new CombinedQuery();

query.Add(new FieldQuery("_shorttemplateid", ShortID.Encode(templateId).ToLowerInvariant()), QueryOccurance.Must);

query.Add(new FieldQuery("_path", ShortID.Encode(parent.ID).ToLowerInvariant())), QueryOccurance.Must);

.....

That's all I wanted to tell about Lucene search index in Sitecore 6. I hope it will help Lucene beginners to better understand the concept and get up to speed with Lucene search index abilities.

Enjoy!

Working with Lucene Search Index in Sitecore 6. Part II - How it works

Here is second part of the Lucene search index overview for Sitecore 6. In this part we'll take a look at configuration settings and talk about how it works.

Sitecore 5 has Lucene engine as well. Let's step one Sitecore version back and see how Lucene works there. In web.config file there is a section /sitecore/indexes that contains Lucene index configuration. When index is configured, it should be added to /sitecore/databases/database/indexes section.

The web database does not have a search index by default. Even if you add it to aforementioned section, it won't work. Why? Because index configuration relies on HistoryEngine functionality. By default the web database does not have it. It's easy to add it though. Just add the HistoryEngine configuration section to the database.

You can find more configuration details from this article on SDN.

This index has the same configuration in Sitecore 6.

In addition to it, Sitecore 6 has a new Lucene index functionality. Which is more reliable and has Sitecore level API on top of Lucene one. In some cases you will still have to use Lucene API. For instance to create range queries.

Configuration settings for new search index located under /sitecore/search section.

The analyzer section defines a Lucene analyzer that is used to analyze and index content.

The categories section is used to categories search results. It's used for content tree search introduced in Sitecore 6. The search box is located right above the content tree in content editor.

The configuration section has indexes definitions with their configurations. An index definition should be created under /sitecore/search/configuration/indexes node.

First two parameters describe the index name and folder name where it should be stored:

<param desc="name">$(id)</param>

<param desc="folder">my_index_folderName</param>

Next setting is the analyzer that should be used for the index:

<Analyzer ref="search/analyzer" />

Lucene StandardAnalyzer covers most of the case scenarios. But it's possible to use any other analyzer if it's needed.

Following setting defines locations for the index:

<locations hint="list:AddCrawler">

It's possible to have multiple locations for one index. Moreover it's even possible to have content from different databases in the same index. Every child of the locations node has its own configuration for a particular part of the content. A name of location node is not predefined. You're welcome to name it the way you want. For example:

<locations hint="list:AddCrawler">

<sdn-site type="Sitecore.Search.Crawlers.DatabaseCrawler, Sitecore.Kernel">

...

</sdn-site>

</locations>

Every location has a database section. It defines indexing database for the location.

Then root section. The database crawler will index content beneath this path.

Next sibling node is the include section. Here it's possible to add templates items of which should be included to the index or excluded from it.

Example:

<include hint="list:IncludeTemplate">

<sampleItem>{76036F5E-CBCE-46D1-AF0A-4143F9B557AA}</sampleItem>

</include>

<include hint="list:ExcludeTemplate">

<layout>{3A45A723-64EE-4919-9D41-02FD40FD1466}</layout>

</include>

It does not make sense to use both of these settings for the one location. Use only one of them.

Next location setting is tags section. Here you can tag indexed content and use it during the search procedure.

Last setting is boost section. Here you have an ability to boost indexed content among other content that belongs to other locations.

And last but not the least, this search index uses the same HistoryEngine mechanism as old one. So, don't forget to copy configuration section from master database to a database where you want to add search index facilities to.

How it all works?

When an action performed on the item, database crawler updates entries in search index for the item. So that information in index is in sync with the one in database. How does it happen if "item:saved", "item:deleted", "item:renamed", "item:copied", "item:moved" do not have event handlers that trigger search index update? Thank to HistoryEngine that was mentioned several times already.

It is HistoryEngine that tracks any changes made to the item and fires appropriate event handler to process it.

IndexingManager is responsible for all operations to the search index. It subscribes to AddEntry event of HistoryEngine and as soon as an entry added to the History table, it triggers a job that updates the search index(es).

In web.config file there are a few settings that belong to indexing functionality.

• Indexing.UpdateInterval - sets the interval between the IndexingManager checking its queue for pending actions. Default value is 5 minutes.
  What does it mean? If for whatever reason pending job was not executed, the IndexingManager will re-run it if it finds it in pending state after 5 minutes pass.
• Indexing.UpdateJobThrottle - sets the minimum time to wait between individual index update jobs. Default value 1 second.
  When some operation is performed on the item, you can see this entry in Sitecore log file:
  INFO Starting update of index for the database 'databaseName' ( pending).
  This setting sets the interval between jobs like this. So that it does not overwhelm all CPU time if you're doing massive change to the items.
• Indexing.ServerSpecificProperties - Indicates if server specific keys should be used for property values (such as 'last updated'). It's off by default.
  This setting is designed for content delivery environments in web farms. As web database is shared, there could be a situation when one server has updated its search indexes and changed History table in the database. Other servers won't update their indexes because HistoryEngine wouldn't indicate there was a change. This setting prevents situations like this.

Well... this is it for now. In next part we will take a look at Sitecore Lucene API and create some search queries with it.

Enjoy!

Working with Lucene Search Index in Sitecore 6. Part I - Why/When to use

How often did you stress yourself thinking on the question "What's the most efficient way to retrieve Sitecore items?"?

Here are possible ways to do it:

• Sitecore API
• Sitecore Query
• SQL custom stored procedures
• Lucene index

Let's take a look at each option briefly.

Sitecore API - the most popular way to get an item. All you need is an item ID and simple call of GetItem() method of database instance will get you the latest item version in current language. If you want an item in specific language, not a problem: GetItem(, ). Want specify version, not a problem either: GetItem(, , ). There are many ways to get an item through Sitecore API. My point is that it's the most easiest way to do it.

Sitecore Query - easy way to get a bunch of items filtered by some criteria. Build a string query, which kinda look like XPath query, and run it on a database instance. Read more about Sitecore Query.

SQL custom stored procedures - we all know how to create a SQL stored procedure. Connections strings already exist in Sitecore solution. All you need is to use some SQL management studio to create a stored procedure for the database. The question is why would you do it if API already exists? It makes sense only if you run complex query with lots of filtering conditions so that SQL will return you only items that you're searching for.

Lucene index - why would I use separate data storage/data index to get an item from Sitecore database? It does not make sense. Agree, for an item it does not but for a collection of items it has a perfect sense to do it. Why? Maybe because performance is much better when you are working with big amount of data. Moreover that's a search index. Which means that data are perfectly organized to be searched.

Let's talk about performance characteristics for each of these options and compare them.

Sitecore API - uses Sitecore data provider to pull out information from the database. If item that we're looking for is cached the call to SQL database is avoided and item gets retrieved from one of Sitecore caches. Using Sitecore caches gives you huge performance benefit. That's why we always worried about caches configuration in Sitecore solution. What if you need to get a number of items by some criteria, let's say template ID and belonging to some category (from here on I'm gonna use this condition for all item retrieval options). You run foreach or for or any other logic that goes through the content tree and checks for a TemplateID property and category field of every single item. If item resides in cache, it will be quick enough but still the code will have to request every single item from the content tree.

Sitecore Query - the picture with Sitecore Query is very similar to Sitecore API but it get's more slower when number of items is growing. Kim has a good article that explains Sitecore Query performance.

NOTE: I'm not talking about fast Sitecore Query introduced in Sitecore 6.

SQL stored procedures - it seems to be a good approach to go with if you're searching for items through the content tree. The thing is there will be only one stored procedure that executes query by specified criteria. Also you will have to consider caching option for retrieved items if you have very deep content tree and items of some specific time are not gathered under one branch. My point is that it can cost you more then the benefits that you will get. I would go with this approach only if I cannot do it with Lucene resources.

Lucene index - again it's search index. It perfectly fits searching options. When you search for items with specified criteria it neither requests an item instance nor runs query over database tables. It goes through the fields that you have in your index and compares their values to your search conditions. The only thing if search query uses custom fields to filter data by, the fields must be added to your index. Otherwise you will get zero results. It's very easy to do though. I will describe it in the next part of the article.

Conclusion: Let's answer those questions in the topic of this part.

Why to use lucene index?

The approach is one of the most (maybe event the best) efficient ones. Search is quick enough that in most cases you don't even need to implement custom caching for the retrieved data.

When to use lucene index?

When you need to run search queries with specific criteria on huge number of data.

Don't forget the rule - the more complex search query gets the slower it works ;).

In next part we will look into Lucene search index introduced in Sitecore 6.

Information about "old" Lucene index (it was introduced in Sitecore 5) you can find here.

Restrict access to advanced media upload options

Sometimes you want to give an access to Advanced Media Upload for editors. When you do it, they get access to all the advanced options that you might not want to share with them.

This package allows you to configure access to advanced media upload options by tighten security on Sitecore items.

After installing the package, you can configure advanced options access for presets items at /sitecore/System/Settings/Media/Presets path.

Download the package.

YouTube Integration source code

For those who's interested, the source code for YouTube data provider is available on Sitecore Trac system. 

Enjoy!

YouTube Integration with Sitecore 6.0

Recently I got a quite interesting task to build an integration between Sitecore CMS and YouTube repository. I finished that task and I'm willing to share my experience with public. I will describe code only partially. Only those parts I consider the most interesting.

We've been thinking about different approaches and have chosen Sitecore Data Provider approach. We decided to have items in CMS that would represent YouTube videos. One of the challenges was to integrate YouTube videos on the fly when an editor wants to add videos from one or other YouTube author.
This data provider works in read-only mode. If you want to extend this solution, you're welcome to do so.
So let's begin....

Templates

I created two templates that will represent root item for YouTube videos and YouTube video item itself. First one is called "YouTube branch" and the other one is "YouTube video".
YouTube branch is going to be a folder for YouTube video items. Unlike usual folders it has a field that determinates videos of what YouTube author to show beneath it.
YouTube video template extends Flash media template and adds two additional fields: Url and Preview. Url is obviously intended for a YouTube video url. Preview is an iframe field that will show us the preview of YouTube video.

Implementation

There is a required list of methods you must implement to get data provider working. Here are those methods:

- GetItemDefinition
- GetItemFields
- GetChildIDs
- GetParentID
- GetItemVersions

I'm not going to describe what they are for. You can find a detailed explanation on SDN.
One method is missing in this list. It's "GetTemplates" method. We can avoid implementing this method if required templates already exist in our data source.
I'm going to use master database as a source for YouTube video items. I created a couple of templates for this approach so that we can skip implementation of the "GetTemplates" method.
Also I made this data provider publish its items by implementing one publishing method:

- GetPublishQueue

This is how constructor looks for the data provider:
public YouTubeDataProvider(string rootTID, string resourceTID, string videoOwnerField, string contentDB)
{
prefix = ToString();
rootTemplateID = rootTID;
resourceTemplateID = resourceTID;
contentDatabase = contentDB;
videoOwnerFieldName = videoOwnerField;
_items = new Hashtable();
}

rootTemplateID contains an ID of YouTube branch template
resourceTemplateID contains an ID of YouTube video template
contentDatabase is a source database for our YouTube provider (it's master database)
videoOwnerFieldName is a field name of YouTube branch template where you specify an owner of YouTube videos.
_items is a temporary cache for our YouTube videos.
Since we need to use some sort of mapping between YouTube videos and Sitecore items, I'm going to use existing functionality of IDTable. That's what we need "prefix" variable for.

Now let's look at implementation of first method:
public override ItemDefinition GetItemDefinition(ID itemId, CallContext context)
{
ItemDefinition itemDef = null;
string itemName = string.Empty;
if (CanProcessYouTubeItem(itemId, context))
{
// method body
}
return itemDef;
}

The first thing we need to think of is how to determine if the item we are getting from data provider is our YouTube item. That's what CanProcessYouTubeItem method for. This method uses IDTable API to find out if an item comes from YouTube rather than SQL database. Here is the code for the method:
bool CanProcessYouTubeItem(ID id, CallContext context)
{
if (IDTable.GetKeys(prefix, id).Length > 0)
{
return true;
}
return false;
}

The same technic is used for GetItemFields, GetParentID and GetItemVersions methods. For GetChildIDs we had to use different code because we had to work with parent item. Here is a code for the method:
public override Sitecore.Collections.IDList GetChildIDs(ItemDefinition itemDefinition, CallContext context)
{
if (CanProcessParent(itemDefinition.ID, context))
{
// method body
}
return null;
}

CanProcessParent method checks if the item is our YouTube branch item. If it indicates that it is then we start processing our child items that are YouTube videos.
This is the code of the method:
bool CanProcessParent(ID id, CallContext context)
{
Item item = ContentDB.Items[id];
bool canProduce = false;
if (item != null && (ID.Parse(rootTemplateID) == item.TemplateID))
{
canProduce = true;
}
return canProduce;
}

After we get our items, we have to fill out their fields to finish our integration process. Information from YouTube video is mapped in GetItemFields method. This is the code:
public override FieldList GetItemFields(ItemDefinition itemDefinition, VersionUri versionUri, CallContext context)
{
FieldList fields = new FieldList();
if (CanProcessYouTubeItem(itemDefinition.ID, context))
{
string originalID = GetOriginalRecordID(itemDefinition.ID);
TemplateItem templateItem = ContentDB.Templates[resourceTemplateID];
if (templateItem != null)
{
YTItemInfo ytItemInfo = GetYTItemInfo(itemDefinition.ID);
if (ytItemInfo != null)
{
foreach (var field in templateItem.Fields)
{
fields.Add(field.ID, GetFieldValue(field, ytItemInfo));
}
}
}
}
return fields;
}

As you can see it's simply iterates through template fields and maps data from YouTube Entry to Sitecore YouTube video item.
We got YouTube video items in our content tree. But we cannot do anything with them on front-end. One option is to add our data provider to web database to get it working on front-end. But what if we make main data provider know about our items and just publish them as a real items. All I need to do is to get list of IDs of our items in GetPublishQueue method.
This is how the code looks:
public override IDList GetPublishQueue(DateTime from, DateTime to, CallContext context)
{
IDList list = new IDList();
foreach (var item in _items)
{
YTItemInfo ytItemInfo = (YTItemInfo) item;
list.Add(ytItemInfo.ItemID);
}
return list;
}

Since I did not use a publish queue table, YouTube items are going to be published every time to run publish operation on them. You can avoid this if you implement AddToPublishQueue and CleanupPublishQueue methods.

Here is a screen dump the way YouTube item looks in Sitecore media library:

This is a sitecore package that will install data provider on your Sitecore 6.0 CMS. I do not described provider configuration in this article. You can find it in /App_Config/Include folder after installing the package.
NOTE: if installation fails, that means that you run into a known issue. Reinstalling the package should solve it.