README

This is the documentation of the Doctrine Lucene Behaviour. This will provide you with full text search in the admin generator filters, and a convenient way to roll out your own powerful search engine.

Knowledge of lucene is not nescesairy, but it is recommended. The jobeet tutorial on searching is an excellent place to start.

Simple Tutorial: Easy admin generator integration
===================
This tutorial will show you how to get fulltext doctrine query with just a few lines.

**1. Install the plugin**

    symfony plugin:install ajDoctrineLucenablePlugin --stability=beta

**2. Configure the plugin**

By default, no configuration is required, but it's good to look into the lucene.yml file.
To configure the plugin: copy the lucene.yml from the ajDoctrineLucenablePlugin/config/lucene.yml to your SF_ROOT/config.
See the yml file for details on the configuration

**2. Enable the lucene behaviour for your model**

    [yaml]
    Project:
      columns:
            id:
              type: integer(4)
              primary: true
              autoincrement: true
            location:
              type: string(255)
              notblank: true
            body:
              type: clob
            name:
              type: string(255)
              notblank: true
      actAs:
            Luceneable:
              fields:
                    - name: unstored
                    - location: unstored
                    - body: unstored

This is a key/value pair, in which the name corresponds with the field (actually the model method for the field, so it can be your custom getFullName() method, then you use the name 'full_name'), and the value corresponds with the Zend Lucene field type (for reference http://framework.zend.com/manual/en/zend.search.lucene.overview.html). This can either be keyword, unindexed, text or unstored. If you're unsure, you will probably want to use unindexed.

Since we 99% of the time only use the lucene index to retrieve document id's, the default value is UnStored. This saves the metadata, but not the data.


**3. (re)build the model**

    symfony doctrine:build --model

**4. Add the filter to the model**

To create a searchable filter for a model, add this to the ProjectFormFilter.class.php:

    [php]
    class ProjectFormFilter extends BaseProjectFormFilter
    {
      public function configure()
      {
            $this->widgetSchema['search'] = new sfWidgetFormFilterInput(array('with_empty' => false));
            $this->validatorSchema['search'] = new sfValidatorPass(array('required' => false));
            $this->useFields(array('search'));
      }

      public function addSearchColumnQuery($query,$field,$value)
      {

        $ids = LuceneSearch::find($value['text'])
          ->fuzzy()
          // search fuzzy?
          ->in($this->getModelName())
          ->setFilterQuery($query);

        return $query;
      }
    }


**5. You're done.**

Example 2: Custom model (recommended)
=========================================

    [yaml]
    Project:
      columns:
        id:
          type: integer(4)
          primary: true
          autoincrement: true
        location:
          type: string(255)
          notblank: true
        body:
          type: clob
        name:
          type: string(255)
          notblank: true
      actAs: [Luceneable]

**In lib/models/doctrine/Project.class.php**:
Here we add to code that is being called when the model is inserted or updated. This example shows how to load and parse a HTML document to the lucene index.
The updateLucene method should always return a Zend_Search_Lucene_Document.

    [php]
    public function updateLucene()
    {
      $doc = Zend_Search_Lucene_Document_Html::loadHTML($this->getBody());
      $doc->addField(Zend_Search_Lucene_Field::Unstored('name', $this->getLocation(), 'utf-8'));
      $doc->addField(Zend_Search_Lucene_Field::Unstored('location', $this->getLocation(), 'utf-8'));
      return $doc;
    }


You can override the parameters from schema.yml from the model by defining a public attribute $luceneSearchFields.

    [php]
    public $luceneSearchFields = array(
        'name' => 'unstored',
        'location' => 'text'
    );


Filters for the Admin Generator
-------
To create a searchable filter for a model (and the admin generator), add this to the **lib/filters/doctrine/FormFilter.class.php**:

    [php]
    class OrganisationFormFilter extends BaseOrganisationFormFilter
    {
      public function configure()
      {
        $this->widgetSchema['search'] = new sfWidgetFormFilterInput(array('with_empty' => false));
        $this->validatorSchema['search'] = new sfValidatorPass(array('required' => false));
        $this->useFields(array('search'));
      }

      public function addSearchColumnQuery($query,$field,$value)
      {
        LuceneSearch::find($value['text'])
          ->fuzzy() // do we want to search fuzzy?
          ->in($this->getModelName())
          ->setFilterQuery($query);
        return $query;
      }
    }


Adding the ->fuzzy() true will mean fuzzy searching (or in fact adding a ~ behind every keyword in the search string.)
This means fuzzy searching. This is great for newbees and noobs, but not so great if you wish to use the cool Zend Lucene Query Language. 



Example 3. Creating your own search engine
======================
The trick in creating a simple search engine is that for each model you have to render a custom partial for that model.

**in search/actions/actions.class.php:**

    [php]
    public function executeSearch(sfWebRequest $request)
    {
        $this->getResponse()->setTitle('Search results for: ' . $request->getParameter('query'));
        $this->searchquery = $request->getParameter('query');
        $query = LuceneSearch::find($query)->fuzzy()->in(array('News','Member','Project','Event','Content','Organisation'));
        $this->results = $query->getRecords();
        $this->hits = $query->getHits();
    }


**in search/templates/searchSuccess.php:**

    [php]
    <?php $count = count($hits);
    if ($count == 0) {
        echo "No results found for '<strong>" . $searchquery . "</strong>':";
    } elseif ($count == 1)
    {
        echo "1 result found for '<strong>" . $searchquery . "</strong>':";
    } else {
        echo sprintf("%d results found",$count) . " for '<strong>" . $searchquery . "</strong>':";
    }

    ?>

    <?php
          foreach ($hits as $hit)
          {
            echo get_partial($hit->model, array(
                'obj' => $results[$hit->model][$hit->pk],
                'pk' => $hit)
            );
          }
    ?>

in the get_partial() line lies the magic, here you can load for each model a seperate template. My template for news looks like this:

**in templates/_News.php**

    [php]
    <?php use_helper('Date'); ?>
    <?php $data = $obj;
    if ($data->is_published):
    ?>
    <div class="contentItem clickableItem">
            <div class="itemDescription">
                    <p>
                            <h1 class="searchCatagory">News</h1><br />
                            <?php echo format_date($data['created_at'], 'EEE dd MMMM yyyy') ?>:
                            <h1><?php echo $data['title'] ?></h1>
                            <?php
                            $abstract = substr($data['abstract'], 0, 180);
                            echo nl2br($abstract) . '... <br />' . link_to('> READ MORE','news/show?id='. $data['id'],array('class'=>'follow'));
               ?>
                    </p>
            </div>
            <div class="itemImage"><img src="/uploads/news/s/<?php echo $data['image']; ?>"></div>
            <div class="borderDiv"> </div>
    </div>
    <?php endif; ?>


Likewise, you can create such a template for each model that you wish to include in your search engine. (in this case for: 'News','Member','Project','Event','Content','Organisation')

Searching
=====

There is a convenient class to search models:

    [php]
    $query = LuceneSearch::find($string)
    ->in('model1','model2','model3');

Results
-------
This method will return the Pk's (primary keys, per model)

    [php]
    $hits = $query->getHits();
    // get an array of lucene hits

    $ids = $query->getPks();
    // get an array of lucene hits in the form:

    array (
    'Model' => array(1,2,3,45,5,6,7),
    'Model2' => array(5,2,1,4)
    )

Connection with the model:
------

    [php]
    $ids = LuceneSearch::find($value['text'])
      //->fuzzy()
      ->in($this->getModelName())
      ->getPksForModel();

    $query = Doctrine_Query::create()
    ->select('m.*')
    ->from('myModel')
    ->andWhereIn('id',$ids);

This will return you the data from the database.

Tasks
=====
This plugin provides a task to (re)index your models, when data has been modified in the database by other means that doctrine.

lucene:reindex
--------------
**usage**
<pre>
symfony lucene:reindex
</pre>
This will use the models defined in lucene.yml (see example in the ajDoctrineLucenablePlugin/config) folder for documentation

<pre>
symfony lucene:reindex Model1 Model2

This will update and optimize Model1 and Model2.
</pre>

lucene:optimize
--------------
**usage**
<pre>
symfony lucene:optimize
</pre>
This will optimize the index file.



Using your own zend framework
===============
By default, the plugin will use the packaged Zend Framework. To disable this behaviour:

**in: lucene.yml**
<pre>
use_packaged_lucene: false
</pre>

Then add in: ProjectConfiguration.class.php:

    [php]
    public function setup()
    {
        // ... load plugins here ... //
        // Here add the listener for the event lucenable.autoload, and connect it to your own
        // ZF autloading method
        $this->dispatcher->connect('luceneable.autoload','ProjectConfiguration::registerZend');
    }

    static protected $zendLoaded = false;

    static public function registerZend()
    {
        if (self::$zendLoaded)
        {
          return;
        }

        set_include_path(sfConfig::get('sf_lib_dir').'/vendor'.PATH_SEPARATOR.get_include_path());
        require_once sfConfig::get('sf_lib_dir').'/vendor/Zend/Loader/Autoloader.php';
        Zend_Loader_Autoloader::getInstance();
        self::$zendLoaded = true;
    }


**Download Zend library** from: http://framework.zend.com/download/current/

To use a minimal zend library, download the entire zip, and from the ZendFramwork-1.x.x/library/Zend copy only these to your **/lib/vendor/Zend** folder:
<pre>
Zend/Loader/*
Zend/Search/*
Zend/Exception.php
Zend/Loader.php
</pre>

Extra Documentation
==============

* [Zend Documentation on the Lucene Query language](http://framework.zend.com/manual/en/zend.search.lucene.query-language.html)
* [Apache Lucene](http://lucene.apache.org/java/2_3_2/queryparsersyntax.html)

	
Things you should know
======================
* lucenable requires you to have a primary key defined, and to call it 'id' (The admin generator will also become funky if you don't do so.)
* The index folder should be writable
* It's useful to optimize the index every now and then. (symfony lucene:optimize)
* This has been tested on small records. For datasets of >100000 perhaps a more industrial solution as java based lucene or solr is recommended.

Classes
=======
The plugin consists of 5 classes
 * **LuceneHandler** (Global class, to handle the autoloading and singleton class for lucene)
 * **LuceneRecord** (Callback methods, for the lucenable listener)
 * **LuceneSearch** (to ease the searching of indexes)
 * **Luceneable** (doctrine Behvaiour)
 * **LuceneableListener** (Doctrine_Record_Listener)

TODO
=====
* make it compatible with i18n (can be accomplished by a method that adds a new record in each language, and by adding a mandatory language field to each record).
* automagickly correct the encoding type
* create a better task to update it
* write tests (performance tests, etc.)
* Evaluate the performance and document the ways to optimize the search results (hand picking a lot of results is not always cool.)