Vertical Search Consultant

What is vertical search?

Vertical search is an app built around a specific industry vertical, such as jobs, real estate, classifieds, automobiles, etc.

Examples of vertical search sites are:

• Homethinking.com (real estate)
• Trulia.com (real estate)
• Indeed.com (jobs)
• SimplyHired.com (jobs)
• ModernSearch.com.au (real estate,jobs)

Vertical search lifecycle

Practically every vertical search engine which aggregates data goes through the following steps:

1. Crawl - Crawl primary data sources
2. Extraction - Extract data into a database
3. Index - Index the data for searching
4. Search - Present data to user in a human-friendly format

Challenges

Some challenges in building and running a vertical search engine include:

• Data sources - not all web sites lend themselves to harvesting
• Developer expertise - broad knowledge of crawling, searching and webapps required
• Maintenance - sites change, and some change often, making maintenance of extraction rules time-consuming

Flexile

I'm the creator and maintainer of Flexile, a vertical search toolkit built specifically for powering aggregation-type vertical search engines. If you're looking to launch a vertical search engine, I recommend you check it out first!

How I can help

I've built a couple of vertical search sites from zero to launch, using either Nutch or a hand-rolled crawler.

I'm intimately familiar with the challenges and pitfalls one usually runs into, and can generally save the team quite abit of time by pre-empting problems.

I've also designed an xml-based site definition fileformat (and a corresponding GUI) which makes it easy to outsource the creation/maintenance of rules which control crawling behavior and data extraction.