词形还原 java [已关闭]

Stanford CoreNLP Java库包含一个稍微占用大量资源的词形分析器，但我在我的笔记本电脑上运行它，RAM<512MB。

要使用它：

1. 下载 jar 文件;
2. 在您选择的编辑器中创建一个新项目/制作一个ant脚本，其中包含您刚刚下载的存档中包含的所有jar文件;
3. 创建一个新的Java，如下所示（基于斯坦福大学网站的代码片段）;

import java.util.Properties;

public class StanfordLemmatizer {

    protected StanfordCoreNLP pipeline;

    public StanfordLemmatizer() {
        // Create StanfordCoreNLP object properties, with POS tagging
        // (required for lemmatization), and lemmatization
        Properties props;
        props = new Properties();
        props.put("annotators", "tokenize, ssplit, pos, lemma");

        // StanfordCoreNLP loads a lot of models, so you probably
        // only want to do this once per execution
        this.pipeline = new StanfordCoreNLP(props);
    }

    public List<String> lemmatize(String documentText)
    {
        List<String> lemmas = new LinkedList<String>();

        // create an empty Annotation just with the given text
        Annotation document = new Annotation(documentText);

        // run all Annotators on this text
        this.pipeline.annotate(document);

        // Iterate over all of the sentences found
        List<CoreMap> sentences = document.get(SentencesAnnotation.class);
        for(CoreMap sentence: sentences) {
            // Iterate over all tokens in a sentence
            for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
                // Retrieve and add the lemma for each word into the list of lemmas
                lemmas.add(token.get(LemmaAnnotation.class));
            }
        }

        return lemmas;
    }
}

克里斯关于斯坦福莱马提器的回答很棒！绝对美丽。他甚至包括一个指向jar文件的指针，所以我不必谷歌搜索它。

但是他的一行代码有语法错误（他以某种方式切换了以“lemmas.add...”开头的行中的结尾右括号和分号），并且他忘记了包含导入。

就NoSuchMethodError错误而言，它通常是由该方法未公开静态引起的，但是如果您查看代码本身（http://grepcode.com/file/repo1.maven.org/maven2/com.guokr/stan-cn-nlp/0.0.2/edu/stanford/nlp/util/Generics.java?av=h），那不是问题所在。我怀疑问题出在构建路径的某个地方（我使用的是Eclipse Kepler，所以配置我在项目中使用的33个jar文件没有问题）。

以下是我对Chris代码的轻微更正，以及一个例子（我向Evanescence道歉，因为他们屠杀了他们完美的歌词）：

import java.util.LinkedList;
import java.util.List;
import java.util.Properties;

import edu.stanford.nlp.ling.CoreAnnotations.LemmaAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.SentencesAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.TokensAnnotation;
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.pipeline.Annotation;
import edu.stanford.nlp.pipeline.StanfordCoreNLP;
import edu.stanford.nlp.util.CoreMap;

public class StanfordLemmatizer {

    protected StanfordCoreNLP pipeline;

    public StanfordLemmatizer() {
        // Create StanfordCoreNLP object properties, with POS tagging
        // (required for lemmatization), and lemmatization
        Properties props;
        props = new Properties();
        props.put("annotators", "tokenize, ssplit, pos, lemma");

        /*
         * This is a pipeline that takes in a string and returns various analyzed linguistic forms. 
         * The String is tokenized via a tokenizer (such as PTBTokenizerAnnotator), 
         * and then other sequence model style annotation can be used to add things like lemmas, 
         * POS tags, and named entities. These are returned as a list of CoreLabels. 
         * Other analysis components build and store parse trees, dependency graphs, etc. 
         * 
         * This class is designed to apply multiple Annotators to an Annotation. 
         * The idea is that you first build up the pipeline by adding Annotators, 
         * and then you take the objects you wish to annotate and pass them in and 
         * get in return a fully annotated object.
         * 
         *  StanfordCoreNLP loads a lot of models, so you probably
         *  only want to do this once per execution
         */
        this.pipeline = new StanfordCoreNLP(props);
    }

    public List<String> lemmatize(String documentText)
    {
        List<String> lemmas = new LinkedList<String>();
        // Create an empty Annotation just with the given text
        Annotation document = new Annotation(documentText);
        // run all Annotators on this text
        this.pipeline.annotate(document);
        // Iterate over all of the sentences found
        List<CoreMap> sentences = document.get(SentencesAnnotation.class);
        for(CoreMap sentence: sentences) {
            // Iterate over all tokens in a sentence
            for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
                // Retrieve and add the lemma for each word into the
                // list of lemmas
                lemmas.add(token.get(LemmaAnnotation.class));
            }
        }
        return lemmas;
    }


    public static void main(String[] args) {
        System.out.println("Starting Stanford Lemmatizer");
        String text = "How could you be seeing into my eyes like open doors? \n"+
                "You led me down into my core where I've became so numb \n"+
                "Without a soul my spirit's sleeping somewhere cold \n"+
                "Until you find it there and led it back home \n"+
                "You woke me up inside \n"+
                "Called my name and saved me from the dark \n"+
                "You have bidden my blood and it ran \n"+
                "Before I would become undone \n"+
                "You saved me from the nothing I've almost become \n"+
                "You were bringing me to life \n"+
                "Now that I knew what I'm without \n"+
                "You can've just left me \n"+
                "You breathed into me and made me real \n"+
                "Frozen inside without your touch \n"+
                "Without your love, darling \n"+
                "Only you are the life among the dead \n"+
                "I've been living a lie, there's nothing inside \n"+
                "You were bringing me to life.";
        StanfordLemmatizer slem = new StanfordLemmatizer();
        System.out.println(slem.lemmatize(text));
    }

}

这是我的结果（我印象非常深刻;它把“'s”当作“is”（有时），并且几乎完美地完成了其他所有事情）：

启动斯坦福词形仪

添加注释器标记化

添加注释器拆分

添加注释器 pos

从 edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger 阅读 POS tagger 模型...完成 [1.7 秒]。

添加注释器引理

[如何，可能，你，是，看到，进入，我的，眼睛，喜欢，打开，门，？，你，领导，我，向下，进入，我的，核心，我，有，变得，所以，麻木，没有，一个，灵魂，我的，精神，'s，睡觉，某处，冷，直到，你，发现，它，那里，和，领导，它，回来，家，你，唤醒，我，向上，里面，呼叫，我的，名字，和，保存，我，从，的， 黑暗，你，有，出价，我的，血，和，它，运行，之前，我，会，成为，撤消，你，拯救，我，从，没有，我，有，几乎，成为，你，成为，带来，我，到，生活，现在，那，我，知道，什么，我，是，没有，你，可以，有，只是，离开，我，你，呼吸，进入，我，和，使，我，真实，冻结，内在，没有，你， 触摸，没有，你，爱，，亲爱的，只有，你，成为，生命，之中，死，我，有，要，活着，一个，谎言，，，有，存在，什么都没有，里面，你，是，带来，我，到，生活，.]