Bon is the first Persian stemmer. It finds the longest substring of a word and removes them from the word. It repeats this process until it there are no more characters to remove. Bon uses the recording technique to transform AXC→AYC, where X is the input string and Y is the string that is transformed. A and C are for the context transformation. Bon uses one dictionary of Persian infinitives, one for ‘mokassar’ words and their singular form, and another dictionary for insensitive words [4]. Because of the numerous exceptions that apply to Persian words, Bon cannot find the correct stem even after all characters are removed through a set of rules [1]. Fig. 2 shows the stemming process in the Bon stemmer.

Four hundred and fifty abstracts of Persian text, which were about computer science and called Persian computer abstract (PCA), were collected to test this stemmer. They used 32 queries to conduct experiments on this collection. Tashakori et al. who made this stemmer [4], claimed that this stemmer could increase recall by 40% on PCAs.

Taghva et al. [10] presented a structural Farsi stemmer in 2005. This stemmer finds the suffixes of a word according the list of Farsi suffixes, which was made by hand using Farsi grammar. If a word has multiple suffixes the stemmer selects the longest suffix [8,9].

The lists of suffixes were categorized into groups like verb-suffixes, plural-noun-suffixes, possessive-noun-suffixes, other-noun-suffixes, and other-suffixes. This categorization helps to remove the Persian of the verbs and suffixes of a noun. Suffixes are stacked where the word is a noun, in accordance with this pattern:

                {Possessive-noun-suffix}{Plural-noun-suffix}{Other-noun-suffix}<Stem>

To specify the suffixes of the input words, the Farsi stemmer applies deterministic finite automata (DFA). The DFA begins at the end of input word and works until the third letter from the front.

To test this stemmer, a collection of 1,647 Farsi documents and 60 queries were used. This stemmer was tested in a vector-based information retrieval system. The test showed an 18% increase in the precision of the information retrieval system. This stemmer was tested on a small collection and so, the effect of the stemmer on larger collections is unknown [10].

Mokhtaripour and Jahanpour [11] studied the Farsi language in-depth and tried to achieve three goals that made the stemmer more accurate. These goals were as follows:

Fig. 3 shows their main operations on the words in Farsi stemmer2:

This stemmer is a rule-based one that has 10 phases. Every phase follows one or more specific grammar rules and after each rule, the length of the rest of the word shouldn’t be less than two characters. This stemmer begins its work by finding the suffixes of the nouns and verbs and removing them. It then tries to remove the prefixes and considers the changes to the words in all of the removal steps. The 10 removal phases are as follows:

There are some Arabic words in Farsi. If the stemmer ensures that a word is Arabic, it applies some of the Arabic rules to find the stem of the word.

A set of 43,680 Farsi documents, which covered various subjects like sport, economics, policy, history, etc. [11], were used to test this stemmer. Twenty-five queries that were related to the documents were also prepared. A classic vector-based system was run without the stemmer in the indexer and the performance of the system was measured. Then, this system was run with the current stemmer and the precision and recall were measured. The results showed that the system has a 46% improvement rate if it used this stemmer [8]. The weakness of this work is that they used a small collection to evaluate the performance of the system and the effect of the stemmer on a bigger collections remains unknown.

Sharifloo and Shamsfard [2] presented a bottom up stemmer that is rule based. Their algorithm has three steps, which are shown in Fig. 4.

In Step 1, they defined which parts of the word were morphemes and which parts were not. In fact, they found the morphological information of all words. They also defined the cluster of each morpheme and the cores in the words. In Step 2, they defined the related rules for the cores that were defined in Step 1. For instance, “ ”(khor) is one core of the word “ ” (mi-xor-am: “I eat”) and it is one of the members of the cluster “ ” (bone mozare: present root), “ ” (am) is one of the members of the cluster “ ” (shenase mozare: present person identifier), and “ ” is a member of the cluster “ ” (mi) [9].

                          
           (Present person identifier + present root + mi)

The result of the second step was a list of cores that extracted all of the rules for the word. In the last step, anti-rule matching, they extracted the anti-rules from the anti-rules repository for every core of the core list. If a core is matched with the word morpheme, it removes it from the core list. As a result, only the correct stems of the words are included in the core list.

This stemmer was evaluated with a limited corpus of the newspaper, Hamshahri. Sharifloo and Shamsfard [2] began their test with 252 rules and 20 anti-rules. This algorithm had a 90.1% correct stemming rate.

Rahimtoroghi et al. [1] used heuristic rules based on the structure of Persian and its exceptions, and created 33 rules to stem the words. They were able to determine some suffix removal rules and used them to stem the words. These rules are shown below in Table 1.

A search engine called Lucene, was used to evaluate this stemmer. This search engine used a probabilistic model called Okapi BM25. This model applied the following formula to calculate the score of document d and query q:

RSV stands for retrieval status value. In this formula, N is the number of whole documents, df_t is the number of documents that contain t, tf_td is the number of occurrences of term t in document d, Ld is the length of the document d, and L_avg is the average length of the document. K1 is set to 1,000 and b is set to 0.75 in the Okapi formula.

Hamshahri collection was used to test this stemmer. The results showed a 4.78% growth rate in the precision of the information retrieval system. The size of the indexed file was decreased, so the speed of the system increased [1].

Perstem, is a morphological analyzer [12]. It uses some regular expression replacements to separate morphemes from stem. Words are first searched for in a hash table. If the word exists in a table, it is the stem and no other work is conducted [11, 13]. If not, then some regular expression substitutions should be carried out. If the stemmer finds a prefix, it puts a +_ between the prefix and the stem and if it finds a suffix, it puts a _+ between the stem and the suffix. Then, the stemmer deletes all prefixes and suffixes, so the rest of the characters are the stem. This stemmer processes 15,000 words in a second on a desktop computer. Jadidinejad et al. [13] tested Perstem. First, they used the Hamshahri corpus and Indri search engine (this test is called “Baseline”), and then Perstem analyzed documents and queries in the Hamshahri corpus and created a new one. This was called Perstem. The experiments showed improvement in recall and precision [14].

In [12], linguistic knowledge and algorithms that support 10 suffixes and almost 2,000 exceptions were used to create the RICesT stemmer. There are two kinds of plural suffixes, “ha” and “an,” which are used to turn singular nouns into plural nouns, like “ghazaha” and “derakhtan,” and “at,” “in,” and “un,” which are mostly used to make plural Arabic words like “sabzijat,” “rohaniyun,” and “sareghin.” It is sometimes necessary to add a letter between words and suffixes [2] (i.e., “chaharpa”+”a” is changed to “chaharpayan”). A singular system produces the single form of the plural nouns by applying the following rules:

A singular stemmer or s-stemmer uses more than 11 rules. The different parts of the s-stemmer are shown in Fig. 5.

Lexical, structural, and syntactical processes are applied to find the stem of a plural noun. The processor that has pattern-based and language methods have four parts, which are as described below.

To find noun lexical knowledge, the system finds a list of grammatical rules that is related to a noun. If there is a new noun, it will be added to the grammatical rules list. The plural nouns will be converted to singular forms via means of natural language processing. The s-stemmer has the following advantages: it can find different forms of a noun and it can automatically classify the text in a large file. Users can personalize the system according to their needs, and create a statistical report of their work. This stemmer was installed in the Regional Information Center for Science and Technology [15].

A statistical Persian stemmer has been presented by Nasiri et al. [17] based on the Bacchin algorithm. Bacchin used statistical methods to distinguish the structures, forms, and grammatical rules of the words in 2002 and completed the implementation of the algorithm in 2005 [18]. He used a complete set of a language words and divided each word into two parts with the first part as a prefix and the second part as a suffix. Each substring was known as the node of a graph, an edge between two nodes, and showed a word of the dataset. Then the links of a substring were analyzed by statistical methods. In the Bacchin algorithm, W shows a dataset of the words, U is used as substrings of the words, and Ω is the combination of two substrings that make a word that exists in a dataset. The formula is shown in (2) [18]:

Two members of Ω have a relation to each other, which is shown by F(z), if and only if they were made of the same word like Z. F(z) partitions Ω into equivalent classes, which are shown by Ω(z):

which are the various states of substrings of a word into two parts. Based on Bayesian rules [5]:

The main purpose of the algorithm is to estimate Pr(yi) for all substrings. Nasiri et al. [17] used the Bacchin algorithm and improved it for Persian stemming. In this stemmer, as shown in Fig. 6, the system finds the lists of the prefixes and suffixes of the words and assigns them weights in the learning phase. It then finds all the substrings of a word in the testing phase and finds the best stems by using the affix list. They used various states of words and found their best stems. They used 800 web documents of ISNA [19] web pages and 32 queries to test this Persian stemmer and because they couldn’t find any search engine for the Farsi language, they used Lucece [20]. They claimed that their stemmer worked well. But, although this model was more complicated than structural approaches, it did not achieve better improvements than structural stemmers. The results showed that structural stemmers work better [1].

Hesamifard and Ghassem-Sani [21] designed a stemmer based on the database’s information. With this method, all of the words and their structure are saved in a database. To find the stem of a word, the input word has to be searched for in a database. If it exists in a database, the stem will be returned, but if the word does not exist in a database, the suffixes and prefixes have to be removed from both sides of the word and the rest of the word has to be searched for in a database again. This algorithm was designed for verbs. There are two different stems for Farsi verbs—past tense and present tense. Adding prefixes and suffixes to them creates all of the verbs of the Farsi language. The rules for making verbs are shown in Table 2. This algorithm does not work right in some states. So, a second version of it was made, called krovetz2. In the krovetz2 algorithm, the verb tenses (past or present) have to be determined first. This solved most of the issues. Krovetz algorithms were tested on 1,000 Farsi sentences. The verbs of the sentences were determined and tested by the krovetz stemmer. Both krovetz and krovetz2 received good results. In krovetz, 223 verbs were tested by the algorithm. This algorithm only made 13 mistakes and krovetz decreased this value to four mistakes. The results showed better results for the krovetz2 algorithm [21]. This algorithm has some weaknesses in that its database needs to be updated and its speed is low [9].

A Persian stemmer that uses a dictionary and a list of prefixes and suffixes has been presented by Dianati et al. [22]. They used word structural similarity to stem the words. It is not necessary to have knowledge about suffixes and prefixes. The steps for the stemming processes are shown in Fig. 7.

In similar word extraction, similar words are extracted from the dictionary based on their structural similarity. In this method, the similarity between words is defined based on numbers and the order of letters in a word. For example, the word “ ” is “ ,” “ ,” and “ .”

Affix lists were used to extract the suffixes and prefixes. The longest affix of a word is removed and the remaining part is the stem.

Five hundred words were selected from the per-Tree-Bank corpus [23] with their stems to test their algorithm and they claimed that 72.8% of the words were correctly stemmed for the Persian language. They used 500 words from the sortedtest.txt dataset to test English words and 71.2% of the words were stemmed correctly. The advantage of this algorithm is its independence of the language and the fact that there is no need to have Persian language morphological knowledge [22].