Six PR for new paddlerec version (#240)

* logistic_regression

* adaptation windows

Co-authored-by: yinhaofeng <[email protected]>
Co-authored-by: yinhaofeng <[email protected]>
Co-authored-by: wuzhihua <[email protected]>
Co-authored-by: tangwei12 <[email protected]>

Author: 4 people

models/match/dssm/data/preprocess.py

@@ -63,7 +63,8 @@
 #划分训练集和测试集
 query_list = list(pos_dict.keys())
 #print(len(query_list))
-#random.shuffle(query_list)
+np.random.seed(107)
+np.random.shuffle(query_list)
 train_query = query_list[:11600]
 test_query = query_list[11600:]
 

models/match/dssm/readme.md

@@ -60,6 +60,12 @@ rm -f dssm%2Fbq.tar.gz
 已经在银行换了新预留号码。      我现在换了电话号码，这个需要更换吗      1
 每个字段以tab键分隔，第1，2列表示两个文本。第3列表示类别（0或1，0表示两个文本不相似，1表示两个文本相似）。
 ```
+在本例中需要调用jieba库和sklearn库，如环境中没有提前安装，可以使用以下命令安装。  
+```
+pip install sklearn
+pip install jieba
+```
+
 ## 运行环境
 PaddlePaddle>=1.7.2
 
@@ -153,11 +159,11 @@ label.txt中对应的测试集中的标签
 4. 退回dssm目录中，打开文件config.yaml,更改其中的参数  
 
 将workspace改为您当前的绝对路径。（可用pwd命令获取绝对路径）  
-将dataset_train中的batch_size从8改为128
-将hyper_parameters中的slice_end从8改为128.当您需要改变batchsize的时候，这个参数也需要随之变化
-将dataset_train中的data_path改为{workspace}/data/big_train
-将dataset_infer中的data_path改为{workspace}/data/big_test
-将hyper_parameters中的trigram_d改为5913
+将dataset_train中的batch_size从8改为128  
+将hyper_parameters中的slice_end从8改为128.当您需要改变batchsize的时候，这个参数也需要随之变化  
+将dataset_train中的data_path改为{workspace}/data/big_train  
+将dataset_infer中的data_path改为{workspace}/data/big_test  
+将hyper_parameters中的trigram_d改为5913  
 
 5.  执行脚本，开始训练.脚本会运行python -m paddlerec.run -m ./config.yaml启动训练，并将结果输出到result文件中。然后启动transform.py整合数据，最后计算出正逆序指标：
 ```

models/match/dssm/run.sh

@@ -13,11 +13,7 @@
 # limitations under the License.
 #!/bin/bash
 echo "................run................."
-python -m paddlerec.run -m ./config.yaml &> result1.txt
-grep -i "query_doc_sim" ./result1.txt >./result2.txt
-sed '$d' result2.txt >result.txt
-rm -f result1.txt
-rm -f result2.txt
+python -m paddlerec.run -m ./config.yaml &> result.txt
 python transform.py
 sort -t $'\t' -k1,1 -k 2nr,2 pair.txt >result.txt
 rm -f pair.txt

models/match/dssm/transform.py

@@ -16,6 +16,20 @@
 import numpy as np
 import sklearn.metrics
 
+filename = './result.txt'
+f = open(filename, "r")
+lines = f.readlines()
+f.close()
+result = []
+for line in lines:
+    if "query_doc_sim" in str(line):
+        result.append(line)
+result = result[:-1]
+f = open(filename, "w")
+for i in range(len(result)):
+    f.write(str(result[i]))
+f.close()
+
 label = []
 filename = './data/label.txt'
 f = open(filename, "r")

models/match/match-pyramid/eval.py

@@ -32,6 +32,20 @@ def eval_MAP(pred, gt):
         return map_value / r
 
 
+filename = './result.txt'
+f = open(filename, "r")
+lines = f.readlines()
+f.close()
+result = []
+for line in lines:
+    if "prediction" in str(line):
+        result.append(line)
+result = result[:-1]
+f = open(filename, "w")
+for i in range(len(result)):
+    f.write(str(result[i]))
+f.close()
+
 filename = './data/relation.test.fold1.txt'
 gt = []
 qid = []
@@ -56,7 +70,7 @@ def eval_MAP(pred, gt):
     pred.append(float(line))
 
 result_dict = {}
-for i in range(len(qid)):
+for i in range(len(pred)):
     if qid[i] not in result_dict:
         result_dict[qid[i]] = []
     result_dict[qid[i]].append([gt[i], pred[i]])

models/match/match-pyramid/readme.md

@@ -55,6 +55,12 @@
 3.关系文件：关系文件被用来存储两个句子之间的关系，如query 和document之间的关系。例如：relation.train.fold1.txt, relation.test.fold1.txt  
 4.嵌入层文件：我们将预训练的词向量存储在嵌入文件中。例如：embed_wiki-pdc_d50_norm  
 
+在本例中需要调用jieba库和sklearn库，如环境中没有提前安装，可以使用以下命令安装。  
+```
+pip install sklearn
+pip install jieba
+```
+
 ## 运行环境
 PaddlePaddle>=1.7.2  
 python 2.7/3.5/3.6/3.7  

models/match/match-pyramid/run.sh

@@ -1,8 +1,4 @@
 #!/bin/bash
 echo "................run................."
-python -m paddlerec.run -m ./config.yaml &>result1.txt
-grep -i "prediction" ./result1.txt >./result2.txt
-sed '$d' result2.txt >result.txt
-rm -f result2.txt
-rm -f result1.txt
+python -m paddlerec.run -m ./config.yaml &>result.txt
 python eval.py

models/match/multiview-simnet/readme.md

@@ -61,6 +61,11 @@ rm -f dssm%2Fbq.tar.gz
 0:358 0:206 0:205 0:250 0:9 0:3 0:207 0:10 0:330 0:164 1:1144 1:217 1:206 1:9 1:3 1:207 1:10 1:398 1:2 2:217 2:206 2:9 2:3 2:207 2:10 2:398 2:2
 0:358 0:206 0:205 0:250 0:9 0:3 0:207 0:10 0:330 0:164 1:951 1:952 1:206 1:9 1:3 1:207 1:10 1:398 2:217 2:206 2:9 2:3 2:207 2:10 2:398 2:2
 ```
+在本例中需要调用jieba库和sklearn库，如环境中没有提前安装，可以使用以下命令安装。  
+```
+pip install sklearn
+pip install jieba
+```
 
 ## 运行环境
 PaddlePaddle>=1.7.2  

models/match/multiview-simnet/run.sh

@@ -14,12 +14,8 @@
 
 #!/bin/bash
 echo "................run................."
-python -m paddlerec.run -m ./config.yaml &>result1.txt
-grep -i "query_pt_sim" ./result1.txt >./result2.txt
-sed '$d' result2.txt >result.txt
-rm -f result1.txt
-rm -f result2.txt
+python -m paddlerec.run -m ./config.yaml &>result.txt
 python transform.py
-sort -t $'\t' -k1,1 -k 2nr,2 pair.txt >result.txt
+sort -t $'\t' -k1,1 -k 2nr,2 pair.txt &>result.txt
 rm -f pair.txt
 python ../../../tools/cal_pos_neg.py result.txt

models/match/multiview-simnet/transform.py

@@ -15,6 +15,20 @@
 import random
 import numpy as np
 
+filename = './result.txt'
+f = open(filename, "r")
+lines = f.readlines()
+f.close()
+result = []
+for line in lines:
+    if "query_pt_sim" in str(line):
+        result.append(line)
+result = result[:-1]
+f = open(filename, "w")
+for i in range(len(result)):
+    f.write(str(result[i]))
+f.close()
+
 label = []
 filename = './data/label.txt'
 f = open(filename, "r")
@@ -31,7 +45,7 @@
 sim = []
 for line in open(filename):
     line = line.strip().split(",")
-    print(line)
+    #print(line)
     line[3] = line[3].split(":")
     line = line[3][1].strip(" ")
     line = line.strip("[")
@@ -50,6 +64,6 @@
 filename = 'pair.txt'
 f = open(filename, "w")
 for i in range(len(sim)):
-    print(i)
+    #print(i)
     f.write(str(query[i]) + "\t" + str(sim[i]) + "\t" + str(label[i]) + "\n")
 f.close()