monad.mmseg.analysis

Class TokenUtils

java.lang.Object

monad.mmseg.analysis.TokenUtils

public class TokenUtils
extends Object

lucene 3.0 从 TokenStream 得到 Token 比较麻烦。

Author:

chenlb 2010-10-7下午10:07:10

Constructor Summary

Constructors

Constructor and Description
TokenUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl	nextToken(org.apache.lucene.analysis.TokenStream input, org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl reusableToken)
static org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl	subToken(org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl oriToken, int termBufferOffset, int termBufferLength)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenUtils

public TokenUtils()

Method Detail

nextToken

public static org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl nextToken(org.apache.lucene.analysis.TokenStream input,
                                                                            org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl reusableToken)
                                                                                     throws IOException

Parameters:

input -

reusableToken - is null well new one auto.

Returns:

null - if not next token or input is null.

Throws:

IOException

subToken

public static org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl subToken(org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl oriToken,
                                                                           int termBufferOffset,
                                                                           int termBufferLength)