v 0.8.0

- Support `Tensorflow Estimator` for large scale data and distributed training.  
- Support  different initializers for different embedding weights by settting `embeddings_initializer` of `SparseFeat`
- Support using pretrained weights to initialize embedding weights and frozen embedding weights. 
- Add new model `FwFM`
- API changes

Author: shenweichen

.github/ISSUE_TEMPLATE/bug_report.md

@@ -18,9 +18,9 @@ Steps to reproduce the behavior:
 4. See error
 
 **Operating environment(运行环境):**
- - python version [e.g. 3.4, 3.6]
- - tensorflow version [e.g. 1.4.0, 1.12.0]
- - deepctr version [e.g. 0.7.1,]
+ - python version [e.g. 3.5, 3.7]
+ - tensorflow version [e.g. 1.4.0, 1.15.0, 2.2.0]
+ - deepctr version [e.g. 0.8.0,]
 
 **Additional context**
 Add any other context about the problem here.

.github/ISSUE_TEMPLATE/question.md

@@ -16,5 +16,5 @@ Add any other context about the problem here.
 
 **Operating environment(运行环境):**
  - python version [e.g. 3.6]
- - tensorflow version [e.g. 1.4.0,]
- - deepctr version [e.g. 0.7.1,]
+ - tensorflow version [e.g. 1.4.0, 1.5.0, 2.2.0]
+ - deepctr version [e.g. 0.8.0,]

CONTRIBUTING.md

@@ -0,0 +1,10 @@
+This project is under development and we need developers to participate in.
+
+If you
+
+- familiar with and interested in ctr prediction algorithms
+- familiar with tensorflow 
+- have spare time to learn and develop
+- familiar with git
+
+please send a brief introduction of your background and experience to [email protected], welcome to join us!

README.md

@@ -18,11 +18,17 @@
 <!-- [![Gitter](https://badges.gitter.im/DeepCTR/community.svg)](https://gitter.im/DeepCTR/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) -->
 
 
-DeepCTR is a **Easy-to-use**,**Modular** and **Extendible** package of deep-learning based CTR models along with lots of core components layers which can be used to easily build custom models.It is compatible with **tensorflow 1.4+ and 2.0+**.You can use any complex model with `model.fit()`and `model.predict()` .
+DeepCTR is a **Easy-to-use**,**Modular** and **Extendible** package of deep-learning based CTR models along with lots of core components layers which can be used to easily build custom models.You can use any complex model with `model.fit()`，and `model.predict()` .
 
-Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html)([Chinese Introduction](https://zhuanlan.zhihu.com/p/53231955))
+- Provide `tf.keras.Model` like interface for **quick experiment**. [example](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#getting-started-4-steps-to-deepctr)
+- Provide  `tensorflow estimator` interface for **large scale data** and **distributed training**. [example](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#getting-started-4-steps-to-deepctr-estimator-with-tfrecord)
+- It is compatible with both `tf 1.x`  and `tf 2.x`.
 
 
+
+
+Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html)([Chinese Introduction](https://zhuanlan.zhihu.com/p/53231955)) and [welcome to join us!](./CONTRIBUTING.md)
+
 ## Models List
 
 |                 Model                  | Paper                                                                                                                                                           |
@@ -40,6 +46,7 @@ Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Star
 |                AutoInt                 | [arxiv 2018][AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/abs/1810.11921)                              |
 |         Deep Interest Network          | [KDD 2018][Deep Interest Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1706.06978.pdf)                                                       |
 |    Deep Interest Evolution Network     | [AAAI 2019][Deep Interest Evolution Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1809.03672.pdf)                                            |
+|                FwFM                    | [WWW 2018][Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://arxiv.org/pdf/1806.03514.pdf)                |
 |                  ONN                  | [arxiv 2019][Operation-aware Neural Networks for User Response Prediction](https://arxiv.org/pdf/1904.12579.pdf)                                                |
 |                 FGCNN                  | [WWW 2019][Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction ](https://arxiv.org/pdf/1904.04447)                             |
 |     Deep Session Interest Network      | [IJCAI 2019][Deep Session Interest Network for Click-Through Rate Prediction ](https://arxiv.org/abs/1905.06482)                                                |

deepctr/__init__.py

@@ -1,4 +1,4 @@
 from .utils import check_version
 
-__version__ = '0.7.5'
+__version__ = '0.8.0'
 check_version(__version__)

deepctr/estimator/__init__.py

@@ -0,0 +1 @@
+from .models import *

deepctr/estimator/feature_column.py

@@ -0,0 +1,52 @@
+import tensorflow as tf
+from tensorflow.python.feature_column.feature_column import _EmbeddingColumn
+
+from .utils import LINEAR_SCOPE_NAME, variable_scope, get_collection, get_GraphKeys, input_layer, get_losses
+
+
+def linear_model(features, linear_feature_columns):
+    if tf.__version__ >= '2.0.0':
+        linear_logits = tf.compat.v1.feature_column.linear_model(features, linear_feature_columns)
+    else:
+        linear_logits = tf.feature_column.linear_model(features, linear_feature_columns)
+    return linear_logits
+
+
+def get_linear_logit(features, linear_feature_columns, l2_reg_linear=0):
+    with variable_scope(LINEAR_SCOPE_NAME):
+        if not linear_feature_columns:
+            linear_logits = tf.Variable([[0.0]], name='bias_weights')
+        else:
+
+            linear_logits = linear_model(features, linear_feature_columns)
+
+            if l2_reg_linear > 0:
+                for var in get_collection(get_GraphKeys().TRAINABLE_VARIABLES, LINEAR_SCOPE_NAME)[:-1]:
+                    get_losses().add_loss(l2_reg_linear * tf.nn.l2_loss(var, name=var.name.split(":")[0] + "_l2loss"),
+                                          get_GraphKeys().REGULARIZATION_LOSSES)
+    return linear_logits
+
+
+def input_from_feature_columns(features, feature_columns, l2_reg_embedding=0.0):
+    dense_value_list = []
+    sparse_emb_list = []
+    for feat in feature_columns:
+        if is_embedding(feat):
+            sparse_emb = tf.expand_dims(input_layer(features, [feat]), axis=1)
+            sparse_emb_list.append(sparse_emb)
+            if l2_reg_embedding > 0:
+                get_losses().add_loss(l2_reg_embedding * tf.nn.l2_loss(sparse_emb, name=feat.name + "_l2loss"),
+                                      get_GraphKeys().REGULARIZATION_LOSSES)
+
+        else:
+            dense_value_list.append(input_layer(features, [feat]))
+
+    return sparse_emb_list, dense_value_list
+
+
+def is_embedding(feature_column):
+    try:
+        from tensorflow.python.feature_column.feature_column_v2 import EmbeddingColumn
+    except:
+        EmbeddingColumn = _EmbeddingColumn
+    return isinstance(feature_column, (_EmbeddingColumn, EmbeddingColumn))

deepctr/estimator/inputs.py

@@ -0,0 +1,47 @@
+import tensorflow as tf
+
+
+def input_fn_pandas(df, features, label=None, batch_size=256, num_epochs=1, shuffle=False, queue_capacity_factor=10,
+                    num_threads=1):
+    if label is not None:
+        y = df[label]
+    else:
+        y = None
+    if tf.__version__ >= "2.0.0":
+        return tf.compat.v1.estimator.inputs.pandas_input_fn(df[features], y, batch_size=batch_size,
+                                                             num_epochs=num_epochs,
+                                                             shuffle=shuffle,
+                                                             queue_capacity=batch_size * queue_capacity_factor,
+                                                             num_threads=num_threads)
+
+    return tf.estimator.inputs.pandas_input_fn(df[features], y, batch_size=batch_size, num_epochs=num_epochs,
+                                               shuffle=shuffle, queue_capacity=batch_size * queue_capacity_factor,
+                                               num_threads=num_threads)
+
+
+def input_fn_tfrecord(filenames, feature_description, label=None, batch_size=256, num_epochs=1, num_parallel_calls=8,
+                      shuffle_factor=10, prefetch_factor=1,
+                      ):
+    def _parse_examples(serial_exmp):
+        features = tf.parse_single_example(serial_exmp, features=feature_description)
+        if label is not None:
+            labels = features.pop(label)
+            return features, labels
+        return features
+
+    def input_fn():
+        dataset = tf.data.TFRecordDataset(filenames)
+        dataset = dataset.map(_parse_examples, num_parallel_calls=num_parallel_calls)
+        if shuffle_factor > 0:
+            dataset = dataset.shuffle(buffer_size=batch_size * shuffle_factor)
+
+        dataset = dataset.repeat(num_epochs).batch(batch_size)
+
+        if prefetch_factor > 0:
+            dataset = dataset.prefetch(buffer_size=batch_size * prefetch_factor)
+
+        iterator = dataset.make_one_shot_iterator()
+
+        return iterator.get_next()
+
+    return input_fn

deepctr/estimator/models/__init__.py

@@ -0,0 +1,12 @@
+from .afm import AFMEstimator
+from .autoint import AutoIntEstimator
+from .ccpm import CCPMEstimator
+from .dcn import DCNEstimator
+from .deepfm import DeepFMEstimator
+from .fwfm import FwFMEstimator
+from .fibinet import FiBiNETEstimator
+from .fnn import FNNEstimator
+from .nfm import NFMEstimator
+from .pnn import PNNEstimator
+from .wdl import WDLEstimator
+from .xdeepfm import xDeepFMEstimator

deepctr/estimator/models/afm.py

@@ -0,0 +1,70 @@
+# -*- coding:utf-8 -*-
+"""
+
+Author:
+    Weichen Shen,[email protected]
+
+Reference:
+    [1] Xiao J, Ye H, He X, et al. Attentional factorization machines: Learning the weight of feature interactions via attention networks[J]. arXiv preprint arXiv:1708.04617, 2017.
+    (https://arxiv.org/abs/1708.04617)
+
+"""
+import tensorflow as tf
+
+from ..feature_column import get_linear_logit, input_from_feature_columns
+from ..utils import deepctr_model_fn, DNN_SCOPE_NAME, variable_scope
+from ...layers.interaction import AFMLayer, FM
+from ...layers.utils import concat_func
+
+
+def AFMEstimator(linear_feature_columns, dnn_feature_columns, use_attention=True, attention_factor=8,
+                 l2_reg_linear=1e-5, l2_reg_embedding=1e-5, l2_reg_att=1e-5, afm_dropout=0, seed=1024,
+                 task='binary', model_dir=None, config=None, linear_optimizer='Ftrl',
+                 dnn_optimizer='Adagrad', training_chief_hooks=None):
+    """Instantiates the Attentional Factorization Machine architecture.
+
+    :param linear_feature_columns: An iterable containing all the features used by linear part of the model.
+    :param dnn_feature_columns: An iterable containing all the features used by deep part of the model.
+    :param use_attention: bool,whether use attention or not,if set to ``False``.it is the same as **standard Factorization Machine**
+    :param attention_factor: positive integer,units in attention net
+    :param l2_reg_linear: float. L2 regularizer strength applied to linear part
+    :param l2_reg_embedding: float. L2 regularizer strength applied to embedding vector
+    :param l2_reg_att: float. L2 regularizer strength applied to attention net
+    :param afm_dropout: float in [0,1), Fraction of the attention net output units to dropout.
+    :param seed: integer ,to use as random seed.
+    :param task: str, ``"binary"`` for  binary logloss or  ``"regression"`` for regression loss
+    :param model_dir: Directory to save model parameters, graph and etc. This can
+        also be used to load checkpoints from the directory into a estimator
+        to continue training a previously saved model.
+    :param config: tf.RunConfig object to configure the runtime settings.
+    :param linear_optimizer: An instance of `tf.Optimizer` used to apply gradients to
+        the linear part of the model. Defaults to FTRL optimizer.
+    :param dnn_optimizer: An instance of `tf.Optimizer` used to apply gradients to
+        the deep part of the model. Defaults to Adagrad optimizer.
+    :param training_chief_hooks: Iterable of `tf.train.SessionRunHook` objects to
+        run on the chief worker during training.
+    :return: A Tensorflow Estimator  instance.
+
+    """
+
+    def _model_fn(features, labels, mode, config):
+        train_flag = (mode == tf.estimator.ModeKeys.TRAIN)
+
+        linear_logits = get_linear_logit(features, linear_feature_columns, l2_reg_linear=l2_reg_linear)
+
+        with variable_scope(DNN_SCOPE_NAME):
+            sparse_embedding_list, dense_value_list = input_from_feature_columns(features, dnn_feature_columns,
+                                                                                 l2_reg_embedding=l2_reg_embedding)
+            if use_attention:
+
+                fm_logit = AFMLayer(attention_factor, l2_reg_att, afm_dropout,
+                                    seed)(sparse_embedding_list, training=train_flag)
+            else:
+                fm_logit = FM()(concat_func(sparse_embedding_list, axis=1))
+
+        logits = linear_logits + fm_logit
+
+        return deepctr_model_fn(features, mode, logits, labels, task, linear_optimizer, dnn_optimizer,
+                                training_chief_hooks=training_chief_hooks)
+
+    return tf.estimator.Estimator(_model_fn, model_dir=model_dir, config=config)