Guidelines for Managers Considering Open Source Enterprise Search

Good Information
Is Hard to Find:

Guidelines for Managers
Considering Open Source Enterprise Search

A Lucid Imagination White Paper

---

Abstract
Enterprise search helps employees, customers, and partners find the most relevant and
timely information, enabling them to make smart, efficient decisions about doing business
with your company. Open source has provided strong benefits in enterprise software such
as operating systems, databases, and middleware, now unleashes value in enterprise
search. Lucid Imagination brings market-leading expertise to open source enterprise
search, and can help any organization quickly design and optimize search solutions based
on Lucene and Solr.

Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 1

---

Table of Contents
Abstract ..................................................................................................................................................................... 1
Introduction and Overview ............................................................................................................................... 3
The Advantages of Open Source ...................................................................................................................... 5
Lower Costs ......................................................................................................................................................... 5
Pay at the Point of Value ................................................................................................................................. 6
Transparent Development ............................................................................................................................ 6
Re-tool the employees, retire the software ............................................................................................. 7
Lower Overall Risk ........................................................................................................................................... 7
About Lucid Imagination .................................................................................................................................... 8
Engagement Scenarios ..................................................................................................................................... 10
Considering Alternatives to Legacy Packaged Search Applications ........................................... 10
Building on In-house Lucene/Solr Expertise ...................................................................................... 12
Next Steps ............................................................................................................................................................. 13
Appendix: About Apache Lucene and Solr ............................................................................................... 14


Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 2

---

Introduction and Overview
Raising the collective intelligence of company employees can make them smarter and more
efficient—but how do you enable them to keep up with the vast, ever-changing amount of
data your organization produces? Many operations seem to be better at creating data than
using it to operate more productively. Using search tools designed for the Web can make it
difficult to find relevant, timely corporate information, mostly because corporate data is
not much like Web data:
• Corporate data can be stored in a variety of different and unstructured formats,
including documents and database records.
• A document’s popularity is not necessarily what makes it useful to a specific search.
• Information may require controlled access, yet still be discoverable to those users
with the appropriate permissions.

Two state-of-the-art, open source search technologies—Lucene and Solr—are available for
free from the Apache Software Foundation. Lucene is a powerful search engine and library;
Solr provides a platform built on top of Lucene that makes it easy to build Lucene-based
applications.1 Rich, flexible text query tools and sophisticated ranking capabilities of
Lucene/Solr enable users to quickly find the most useful documents or records.
Either of these full-featured technologies delivers excellent performance, relevancy
ranking, and scalability. They are used today by thousands of organizations, powering
substantial and diverse search applications for AOL, CNET, Comcast Interactive Media, IBM,
Netflix, LinkedIn, MySpace, and many others. For these companies, Lucene/Solr solutions
regularly index and search hundreds of millions of documents with subsecond response
time, all without incurring any licensing fees.
These solutions excel at quickly and effectively searching large volumes of unstructured
text—documents or other records containing freeform text—and returning results based


1 Most organizations use Solr today as their search development platform. Because Lucene serves as the core of
Solr’s search capabilities, this paper refers to them as Lucene/Solr. For more information about these technologies,
see the Appendix.
Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 3

---

on how well they match the user’s query. At most companies, this means digesting and
searching through dozens of different file formats—including documents, spreadsheets,
presentations, e-mail, and records stored in databases, to name just a few—and delivering
relevant results to authorized users. Incremental update capabilities mean that
Lucene/Solr searches can track document collections easily as they grow and change,
finding information nearly as fast as it is created.
Solr can speedily facet, or categorize, data and search results based on specific field values.
An excellent example of this function is Zappos.com, the popular shoe e-tailer, where users
can quickly refine searches based on product criteria such as price or features.
For most application development teams, building a search application is not an everyday
project. By definition, enterprise search technology processes unstructured data, which can
change frequently. Expert guidance on architectural considerations, such as index
optimization, result relevance, deployment configuration, and retrieval performance can
make a tremendous difference in deploying a successful solution. By taking advantage of
expert, experienced personnel to assist with application design, development, and
deployment, organizations can leverage the full benefit of Lucene/Solr search technologies
without the cost of licensing proprietary software.
For these reasons, Lucid Imagination provides commercial-grade support, training, and
professional consulting services that are essential to designing and installing successful
enterprise applications.
This paper is intended for business decision makers who are considering options for
powerful, flexible enterprise search solutions. It provides guidelines for understanding:
• Advantages of open source software, including ways it can lower costs and risks,
• Why Lucid Imagination’s service and support is a key ingredient in achieving successful
Lucene/Solr solutions,
• Engagement scenarios—the types of situations where Lucid Imagination can help, and
• The capabilities of Lucene/Solr, which are provided in an appendix.


Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 4

---

The Advantages of Open Source
Open Source has changed the IT landscape. Gartner says 85 percent of polled companies
are already using open source software, calling the use of open source software
“pervasive.”2 Most organizations are now familiar with free and open source products such
as Linux, MySQL, Apache, and SugarCRM, because of the many benefits, including:
• Lower costs
• Pay at the point of value
• Transparent development
• Control and flexibility – investing in people instead of licenses
• Lower overall risk

With Lucene/Solr’s broad, successful adoption across markets and deployments, these
advantages are now available for enterprise search applications. Let’s take a closer look at
how open source pays off.
Lower Costs
While proprietary software vendors must try to recover their development costs, this is not
the case with open source software, because it does not have capital costs associated with
source code IP. The cost of talent is less, too. Community development, adherence to
standards, and lower barriers to adoption all help increase the number of developers who
become proficient in the use of a product or technology. Together, these factors combine to
reduce upward pricing pressure.
The high license fees associated with proprietary and closed source development can
discourage developers and customers from adopting a product or technology. In contrast,
open source communities help lower costs by encouraging participation and allowing
anyone to download the source code and try it out. Most open source communities release


2 http://www.theregister.co.uk/2008/11/18/gartner_open_source/
Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 5

---

updated binaries on a periodic basis, so users can easily try the software on their own
timetables.
Many commercial solutions combine proprietary software with service and support, and
customers may believe that buying a software license is sufficient to get a search
application up and running. In most cases, however, the technology’s purchase price makes
up less than half of the implementation cost, with the balance going to services. Both open
source and proprietary software usually require a significant amount of customization,
which means some service and support costs are inevitable.
Pay at the Point of Value
Open source project code is freely available for any use. If a company can become proficient
with the code, it can make productive use the code at any phase from evaluation to
production. Only in those areas where an open source customer sees value—for support
and integration services, or for additional functionality or expertise—does money need to
be spent. There are no restrictions on when open source software can be used.
In contrast, proprietary products typically must be purchased before they can be used, or
in some cases, even evaluated. Some vendors offer evaluation or trial versions, but these
often have reduced functionality or restrictive licenses. Because the software must be
purchased before the customer can see any value from the product, return on investment is
delayed.
Transparent Development
Community-developed software enables everyone to see what is being built and which
features are included as early as possible. Developers and customers do not need to wait
for a vendor to publish a roadmap or product launch to know what is being readied for
release. As a result, prospective users can make better, faster, and more informed decisions
relating to their software infrastructure.
Compare this to proprietary software, where customers have little if any insight into
upcoming products until very late in the product life cycle. This is typically no sooner than
the software’s beta release, when it is too late to provide input on features and
functionality. This delays assessment and adoption of innovations.

Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 6

---

Re-tool the employees, retire the software
In this tough economic climate, managers who own budgets need to review every expense
with a critical eye. Many software applications that made sense a few years back may have
out-lived their intended fit to business needs.
Any application development effort generates significant learning. The work of
development imbues it requires the expertise of in-house developers with deep knowledge
and understanding of the company, its IT infrastructure, culture, and usage requirements.
Given that software applications must keep up with an organization’s changing goals and
requirements as the needs of its market and constituents evolve, the expertise which the
technical staff develops becomes is a vital competitive asset.
This is key corollary benefit of the open source model: by retiring old software packages
and investing in staff expertise, companies combine innovative technology with their most
valuable asset – their people, establishing vital competitive advantage.
Companies who leverage savings from not purchasing software licenses to build
development talent in-house reduce the cost of addressing inevitable change. What’s more,
increasing a technical team’s ability to translate company business objectives into
technology solutions increases the likelihood that the software they build will continue to
fit that inevitable change. This is particularly true for an enterprise search solution. What’s
more, compared to closed source implementations, in-house developers can work with
open source code and supplement additional functions or expertise by relying on the
community and marketplace of readily available resources – again capturing unique
competitive advantage.
Supplementing open source development with training, consulting, and reliable support
from established industry experts reinforces a company’s competitive advantage – with the
control and flexibility needed to survive and thrive.
Lower Overall Risk
Vendors use proprietary interfaces and components to lock in customers. However, the
source code for open source software is freely available and widely supported by the
community, based on standardized, free public interfaces. If a commercial vendor goes out
of business (or is purchased by another), or tries to increase fees for a commercial product,
Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 7

---

open source vendors may be able to step in to meet the needs of customers at market-
competitive prices.
Open source software can reduce security and operational risks, too. Widely used open
source software is essentially under constant peer review. Technical or security issues,
once exposed in the community, are readily addressed, resulting in a safer and more
reliable product.
About Lucid Imagination
The benefits of open source have unlocked tremendous value in many software categories:
Red Hat’s Enterprise Linux in operating systems, MySQL in database software, Sugar in
CRM software—all have benefited from matching the efficiencies of open source with deep,
robust commercial resources to ensure successful applications. Today, Lucid Imagination’s
capabilities and expertise brings that same approach to unlocking enterprise search with
Lucene and Solr.
Lucid Imagination’s mission is to enable customers to achieve business objectives for
optimal search performance and accuracy, with lower total cost of ownership and faster
time to market. The company’s founding team consists of many key contributors and
committers to the Lucene/Solr project, as well as other experts in enterprise search
application development. Our skills, acquired across hundreds of deployments, including
best practices and technical know-how, can enhance and optimize any phase of an open
source search implementation.
Lucid Imagination’s team has a deep understanding of indexing, which is the foundation of
any search solution; it captures all the content and location of searched documents for
quick lookup, much as a book index does. We have broad experience indexing:
• Documents of widely varying sizes and formats within a very large collection,
• Documents with diverse metadata requirements, and
• Multilingual documents.

The team is also skilled at applying business rules such as boosting documents and fields,
indexing dates, or other attributes of terms and data. Lucid Imagination has developed best
practices for indexing and metadata management, and can help establish and refine
policies to meet business and technical search requirements, such as:
Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 8

---

• How and when to add documents to an index,
• Removing documents from an index,
• Results relevancy and document/data findability
• Undeleting documents, and
• Batch and real-time updates.

The Lucid Imagination team has extensive experience with large-scale search applications,
including engagements with:
• Large collections—more than one billion documents,
• High query volumes and large user populations,
• High document growth rates,
• Distributed indexing and searching,
• Replication and high availability, and
• Cloud environments.

In addition to fine-tuning search technology machinery, the Lucid Imagination team has
significant expertise in natural language processing, which optimizes the interaction of
compute resources with human-created content. Key considerations include:
• Developing structured methods for characterizing how well a set of results meets user
needs,
• Establishing a tradeoff between overall net gain in the quality of results across the whole
application, versus a single improvement for one query or user, and
• Improving the ability to find accurate answers by leveraging a balanced mix of content
analysis and query interpretation algorithms.
The breadth of expertise offered by Lucid is available in a variety of forms suited to a range
of different business needs and deployment requirements. This enables customers to
create even more powerful and successful search applications.

Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009

Page 9

---