Think of Search Engine Replacement… as an Upgrade!

Planning an enterprise search engine replacement is much like replacing a cell-phone from 10 years ago with the latest model. There is not just a quantitative difference (faster, more memory) but also a qualitative difference (applications that were simply not even conceivable a decade ago) in what you need to think about.

And while the thought of change is daunting, we’ve created a checklist that should let you realize new gains with your new enterprise search engine replacement.

Reconsider Your Enterprise Search Needs

Before beginning the process of replacing your enterprise search software, consider these questions:

• What use case will your search application serve? Different search use cases have different needs, depending on the desired result. Identifying the use case informs many other aspects of the search engine replacement process, including identifying challenges that need to be overcome and illuminating not-thought-of steps.  
• Which information systems should be indexed? Once you know your use case, it should be easier to identify which data sources should be indexed (for example, a workplace use case might want to bring knowledge articles, community forums, and service cases together to create a multi-dimensional view).
• Which content types should be indexed? And from there, the decision-making process should flow into the types of content to be ingested and indexed for search. This can include not just traditional documents (as we discussed in our previous piece), but also unstructured content and even conversational content. 
• What are you trying to search? Some contemporary enterprise search systems are fairly broad, and can cover images, video, and audio using natural language processing (NLP) and image recognition software, as well as more traditional documents.
• How do you currently manage your corpus (i.e., collection of content)? Are documents contained within a file system, a content management system, or a more generalized service infrastructure such as SharePoint? To the extent possible, your search engine should manage the metadata, models, and indexes of that corpus independent of how those documents are stored and retrieved.
• Is your search engine services oriented? By exposing the engine as a set of APIs (cloud-based web endpoints), a services-oriented search engine can bridge multiple applications and platforms.
• Does your search solution offer recommendations? A recommendation engine can expose users to new content they weren’t previously aware of. Different recommendation engine configurations have various tradeoffs, the main one being how much data you have available to provide it.

As you start planning the conversion process, there are additional caveats that you should be aware of:

• In a number of older systems, there were likely several customizations that your organization made to improve search quality with your older system. Features such as stemming (building indexes on variations of words), ranking algorithms, the taxonomy used for classification and similar product enhancements more than likely are now available out of the box for many new products, and as such the need of manually porting them may prove unnecessary.
• Moreover, attempting to port these customizations manually may actually prove counterproductive or even dangerous, as it could quite possibly degrade performance compared to what’s available in the newer products.
• Does it make sense to build (or redesign) a custom search engine? Ten years ago, it might have. Today, there is very little benefit to doing so. Enterprise search software has become more powerful, but it has also become more complex, as machine learning, semantics, neural networks and natural language processing (NLP) have all shifted from being largely algorithmic in nature to being more artificial intelligence and heuristic focused. This has also reduced price-points dramatically, especially in conjunction with newer hardware and platforms, to the extent that building an enterprise search engine today from scratch is just not cost effective.
• Another consideration to keep in mind is that hardware and operating system options have increased as well. Cloud hosting of both your corpus and the search engine is much more feasible today, which also provides the advantage of scaling of memory and compute resources. 
• Similarly, when considering an insight engine, finding ones that provide front-end web components for everything from Google-style search pages to intelligent self-service portals is becoming ever more feasible. As with not reinventing the wheel on the server, taking advantage of pre-existing web components can get you up and running on a new search engine within hours rather than weeks.

Migrating From One Search to Another

Given all of this, there are six key steps in migrating to a more contemporary search engine:

Step 1. Define the Use Case

• Identify Your Audience. Who will be using it, how will they be using it, and what expectations they will have for successful use?
• Identify Content Repositories and Define Applicable Content. Both the content and the container the content lives in can affect indexing. Are you working just with text documents, email, PDFs, images, video or audio with or without transcripts? Are you looking for document-level search or the ability to search within documents?
• Capture Requirements. Determine what kind of search is required: Are you looking for speed, ease of training, greater relevance, ease of use? What are your priorities, and what can you afford to sacrifice?
• Outline Benefits Driven By Use Case. What’s the purpose of the migration, beyond potential end of product life issues? What are the business benefits to the approach you’re taking, from faster access to data to more utility in analysis to improved customer retention?
• Assess Value From the Benefits. Can you translate these benefits into quantifiable measures? If you had to make a presentation to your company’s leadership, could you justify the expense in terms of return on investment?

Step 2. Select a Solution

• Establishing the Budget. If money was no obstacle, everyone would get the best search experience they could. Since it is, one of the first issues you need to resolve is what are you willing to spend. This may vary as requirements are ascertained, so you should establish the budget only once you have an assessment about what value the enterprise search solution provides to your organization.
• Compare Requirements with Solution Capabilities. There is no perfect search solution, because organizational needs differ considerably. Compare what your requirements are with what vendors or open source projects offer (keeping in mind that the latter may not necessarily have the technical support of the former). 
• Identify The Best Solution For Your Needs. If at all possible, try out the product in a test environment with your most-likely users and compare results that way. Talk with vendors about evaluation versions, and identify whether these test products offer a sufficiently full experience.
• Negotiate Pricing and Licensing. As with requirements, most vendors may offer different pricing structures based upon product needs and exposed functionality. They may also offer different licensing agreements based upon core CPU/GPU usage, seat usage or similar metric.

Step 3. Design the Experience

• Determine Project Roadmap. Identify the schedule, actors, goals and expectations for the project at hand.  This usually involves migrating existing content, creating the user experience (which will likely be different from something out of the box), and determining the criteria for success.
• Identify Key Contributes and Potential Blockers. This establishes roles, determines the stakeholders in the project, and highlights potential risk to the project.
• Map Requirements to Solution Capabilities. This is a more thorough analysis than in Step 3, though may build on it. In effect, this identifies the various activities that need to be facilitated to implement the solution given the requirements.
• Outline Technical Integration Points. This determines how the product will fit in with other applications within the organization, both current and potentially in the future.
• Establish User Experience. This identifies the interfaces through which users will interact with the search application, typically by the creation of wireframes.

Step 4. Implement

• Index the Content. The search engine’s ingestion system reads the content, and then builds the relevant indexes that makes it easier to search for that content. This typically involves natural language processing, semantics, and machine learning systems. Note that the content may or may not need to be stored with the search information and additionally may or may not be enhanced by the ingestion process.
• Implement User Experience. This converts the wireframes into working application user interfaces, and also helps to identify inferior approaches on those interfaces for subsequent fixes.
• Configure or Integrate Relevance Logic. This is the process of creating (or training) rules for determining relevance, semantic meaning, and context with the documents. It may be done in conjunction with the indexing, or may be handled as a separate process.

Step 5. Validate

• Perform User Acceptance Testing. Put your users in front of mid-to-late iterations of the implemented systems, tell them to break the system, then use that to figure out how to strengthen the user experience and minimize potential corruption.
• Have Subject Matter Experts Optimize Relevance. Your subject matter experts will know whether the search engine is in fact providing the best potential responses, and these SMEs should work closely with the development team to insure that relevance parameters can be fine-tuned for optimal results.
• Implement and Review Analytics. Dashboards play a significant role here, identifying performance considerations, complexity of queries, relevance measures, and potential bottlenecks.

Step 6. Deploy

• Pull the Big Red Switch. Move content, rules, and other code and content to the production environment, making it available for the target audience.
• Enable End User Experience. This covers a fair amount of area, from doing user training to enabling technical support to activating chat bots and other autonomous artificial intelligence components.
• Tune Relevancy To Reflect Usage. As users become more familiar with the environment, they will identify areas where relevancy isn’t as targeted as they might have hoped. These users should be relied upon to provide opportunities to better tweak relevancy scores, improve semantics, and reinforce AI learning.
• Iteratively Test Your Assumptions. Deployment is not the end of the story. Once deployed, implementers and managers should go back to the base audience periodically and provide them with comparison scenarios to determine how closely a given use case scenario has worked out in practice. This can incorporate such methods as A/B or multivariate testing to test a hypothesis by comparing two different search pipelines or by testing multiple versions of an online experience..

Search is one of the most important functions in any organization, for a very simple reason. The goal of search is not to find a document … it’s to find an answer. 

Search is the mechanism that’s used to access the accrued wisdom of the many people who have worked before you, along with potentially broader access to external information that ties into what you’re doing. It may be the means that your customers best access the most appropriate products for them with the most relevant information from those products. It also makes this information available based upon criteria as diverse as when the content was created to the skill level that said information is best suited for to the most relevant content topically. 

Search also plays a role in managing information overload. In a world where content has become the primary tool to create a dialogue and relationship with customers, search is essential to helping to win attention.

Migrating over to a contemporary search solution should not be an onerous challenge, especially as the benefits that come from such solutions—improved relevancy, better semantic support, reinforcement learning that makes it easier for such search to adapt to the users and so forth—should easily outweigh the costs and pains of that action.

Good luck, and may you find what you seek!

Dig Deeper

One of the most important steps of enterprise search engine replacement is evaluating options on the market. If you’re wondering where to started when looking to evaluate different enterprise search platform options on the market? Check out our Buyers Guide for Enterprise Search Platforms for the inside scoop.