如何快速正确分词,对于SEO来说,是提取tags聚合,信息关联的好帮手。
目前很多分词工具都是基于一元的分词法,需要词库来辅助。
通过对Google黑板报第一章的学习,如何利用统计模型进行分词。
本方法考虑了3个维度
凝聚程度:两个字连续出现的概率并不是各自独立的程度。例如“上”出现的概率是1×10^-5,”床”出现的概率是1×10^-10,如果这两个字的凝聚程度低,则”上床”出现的概率应该和1×10^-15接近,但是事实上”上床”出现的概率在1×10^-11次方,远高于各自独立概率之积。所以我们可以认为“上床”是一个词。
左邻字集合熵:分出的词左边一个字的信息量,比如”巴掌”,基本只能用于”打巴掌”,“一巴掌”,“拍巴掌”,反之”过去”这个词,前面可以用“走过去”,“跑过去”,“爬过去”,“打过去”,“混过去”,“睡过去”,“死过去”,“飞过去”等等,信息熵就非常高。
右邻字集合熵:分出的词右边一个词的信息量,同上
1
2
3
4
5
6
7
8
9
|
#!/bin/sh python . /splitstr .py > substr.freq python . /cntfreq .py > word.freq python . /findwords .py > result sort -t " " -r -n -k 2 result > result. sort |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
import math def compute_entropy(word_list): wdict = {} tot_cnt = 0 for w in word_list: if w not in wdict: wdict[w] = 0 wdict[w] + = 1 tot_cnt + = 1 ent = 0.0 for k,v in wdict.items(): p = 1.0 * v / tot_cnt ent - = p * math.log(p) return ent def count_substr_freq(): fp = open ( "./video.corpus" ) str_freq = {} str_left_word = {} str_right_word = {} tot_cnt = 0 for line in fp: line = line.strip( '\n' ) st = line.decode( 'utf-8' ) l = len (st) for i in range (l): for j in range (i + 1 ,l): if j - i 0 : left_word = st[i - 1 ] else : left_word = '^' if j < l - 1 : right_word = st[j + 1 ] else : right_word = '%' str_left_word[w].append(left_word) str_right_word[w].append(right_word) tot_cnt + = 1 for k,v in str_freq.items(): if v > = 10 : left_ent = compute_entropy(str_left_word[k]) right_ent = compute_entropy(str_right_word[k]) print "%s\t%f\t%f\t%f" % (k,v * 1.0 / tot_cnt,left_ent,right_ent) if __name__ = = "__main__" : count_substr_freq() |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
def count_freq(): word_freq = {} fp = open ( "./substr.freq" ) tot_cnt = 0.0 for line in fp: line = line.split( '\t' ) if len (line) < 2 : continue st = line[ 0 ].decode( 'utf-8' ) freq = float (line[ 1 ]) for w in st: if w not in word_freq: word_freq[w] = 0.0 word_freq[w] + = freq tot_cnt + = freq while True : try : x,y = word_freq.popitem() if x: freq = y * 1.0 / tot_cnt print "%s\t%f" % (x.encode( 'utf-8' ),freq) else : break except : break if __name__ = = "__main__" : count_freq() |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
|
def load_dict(filename): dict = {} fp = open (filename) for line in fp: line = line.strip( '\n' ) item = line.split( '\t' ) if len (item) = = 2 : dict [item[ 0 ]] = float (item[ 1 ]) return dict def compute_prob( str , dict ): p = 1.0 for w in str : w = w.encode( 'utf-8' ) if w in dict : p * = dict [w] return p def is_ascii(s): return all ( ord (c) < 128 for c in s) def find_compact_substr( dict ): fp = open ( "./substr.freq" ) str_freq = {} for line in fp: line = line.decode( 'utf-8' ) items = line.split( '\t' ) if len (items) < 4 : continue substr = items[ 0 ] freq = float (items[ 1 ]) left_ent = float (items[ 2 ]) right_ent = float (items[ 3 ]) p = compute_prob(substr, dict ) freq_ratio = freq / p if freq_ratio > 5.0 and left_ent > 2.5 and right_ent > 2.5 and len (substr) > = 2 and not is_ascii(substr): print "%s\t%f" % (substr.encode( 'utf-8' ),freq) if __name__ = = "__main__" : dict = load_dict( './word.freq' ) find_compact_substr( dict ) |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
视频 0.000237 轴承 0.000184 北京 0.000150 中国 0.000134 高清 0.000109 搞笑 0.000101 新闻 0.000100 上海 0.000100 美女 0.000092 演唱 0.000085 音乐 0.000082 —— 0.000082 第二 0.000080 少女 0.000078 最新 0.000074 广场 0.000070 |