Skip to content

Commit 64291fc

Browse files
authored
Merge pull request #374 from yahoo/leewyang_oath_oss
Update READMEs per guidelines
2 parents 7b36521 + 5f1a3c3 commit 64291fc

File tree

3 files changed

+118
-23
lines changed

3 files changed

+118
-23
lines changed

Code-of-Conduct.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Oath Open Source Code of Conduct
2+
3+
## Summary
4+
This Code of Conduct is our way to encourage good behavior and discourage bad behavior in our open source community. We invite participation from many people to bring different perspectives to support this project. We pledge to do our part to foster a welcoming and professional environment free of harassment. We expect participants to communicate professionally and thoughtfully during their involvement with this project.
5+
6+
Participants may lose their good standing by engaging in misconduct. For example: insulting, threatening, or conveying unwelcome sexual content. We ask participants who observe conduct issues to report the incident directly to the project's Response Team at [email protected]. Oath will assign a respondent to address the issue. We may remove harassers from this project.
7+
8+
This code does not replace the terms of service or acceptable use policies of the websites used to support this project. We acknowledge that participants may be subject to additional conduct terms based on their employment which may govern their online expressions.
9+
10+
## Details
11+
This Code of Conduct makes our expectations of participants in this community explicit.
12+
* We forbid harassment and abusive speech within this community.
13+
* We request participants to report misconduct to the project’s Response Team.
14+
* We urge participants to refrain from using discussion forums to play out a fight.
15+
16+
### Expected Behaviors
17+
We expect participants in this community to conduct themselves professionally. Since our primary mode of communication is text on an online forum (e.g. issues, pull requests, comments, emails, or chats) devoid of vocal tone, gestures, or other context that is often vital to understanding, it is important that participants are attentive to their interaction style.
18+
19+
* **Assume positive intent.** We ask community members to assume positive intent on the part of other people’s communications. We may disagree on details, but we expect all suggestions to be supportive of the community goals.
20+
* **Respect participants.** We expect participants will occasionally disagree. Even if we reject an idea, we welcome everyone’s participation. Open Source projects are learning experiences. Ask, explore, challenge, and then respectfully assert if you agree or disagree. If your idea is rejected, be more persuasive not bitter.
21+
* **Welcoming to new members.** New members bring new perspectives. Some may raise questions that have been addressed before. Kindly point them to existing discussions. Everyone is new to every project once.
22+
* **Be kind to beginners.** Beginners use open source projects to get experience. They might not be talented coders yet, and projects should not accept poor quality code. But we were all beginners once, and we need to engage kindly.
23+
* **Consider your impact on others.** Your work will be used by others, and you depend on the work of others. We expect community members to be considerate and establish a balance their self-interest with communal interest.
24+
* **Use words carefully.** We may not understand intent when you say something ironic. Poe’s Law suggests that without an emoticon people will misinterpret sarcasm. We ask community members to communicate plainly.
25+
* **Leave with class.** When you wish to resign from participating in this project for any reason, you are free to fork the code and create a competitive project. Open Source explicitly allows this. Your exit should not be dramatic or bitter.
26+
27+
### Unacceptable Behaviors
28+
Participants remain in good standing when they do not engage in misconduct or harassment. To elaborate:
29+
* **Don't be a bigot.** Calling out project members by their identity or background in a negative or insulting manner. This includes, but is not limited to, slurs or insinuations related to protected or suspect classes e.g. race, color, citizenship, national origin, political belief, religion, sexual orientation, gender identity and expression, age, size, culture, ethnicity, genetic features, language, profession, national minority statue, mental or physical ability.
30+
* **Don't insult.** Insulting remarks about a person’s lifestyle practices.
31+
* **Don't dox.** Revealing private information about other participants without explicit permission.
32+
* **Don't intimidate.** Threats of violence or intimidation of any project member.
33+
* **Don't creep.** Unwanted sexual attention or content unsuited for the subject of this project.
34+
* **Don't disrupt.** Sustained disruptions in a discussion.
35+
* **Let us help.** Refusal to assist the Response Team to resolve an issue in the community.
36+
37+
We do not list all forms of harassment, nor imply some forms of harassment are not worthy of action. Any participant who *feels* harassed or *observes* harassment, should report the incident. Victim of harassment should not address grievances in the public forum, as this often intensifies the problem. Report it, and let us address it off-line.
38+
39+
### Reporting Issues
40+
If you experience or witness misconduct, or have any other concerns about the conduct of members of this project, please report it by contacting our Response Team at [email protected] who will handle your report with discretion. Your report should include:
41+
* Your preferred contact information. We cannot process anonymous reports.
42+
* Names (real or usernames) of those involved in the incident.
43+
* Your account of what occurred, and if the incident is ongoing. Please provide links to or transcripts of the publicly available records (e.g. a mailing list archive or a public IRC logger), so that we can review it.
44+
* Any additional information that may be helpful to achieve resolution.
45+
46+
After filing a report, a representative will contact you directly to review the incident and ask additional questions. If a member of the Oath Response Team is named in an incident report, that member will be recused from handling your incident. If the complaint originates from a member of the Response Team, it will be addressed by a different member of the Response Team. We will consider reports to be confidential for the purpose of protecting victims of abuse.
47+
48+
### Scope
49+
Oath will assign a Response Team member with admin rights on the project and legal rights on the project copyright. The Response Team is empowered to restrict some privileges to the project as needed. Since this project is governed by an open source license, any participant may fork the code under the terms of the project license. The Response Team’s goal is to preserve the project if possible, and will restrict or remove participation from those who disrupt the project.
50+
51+
This code does not replace the terms of service or acceptable use policies that are provided by the websites used to support this community. Nor does this code apply to communications or actions that take place outside of the context of this community. Many participants in this project are also subject to codes of conduct based on their employment. This code is a social-contract that informs participants of our social expectations. It is not a terms of service or legal contract.
52+
53+
## License and Acknowledgment.
54+
This text is shared under the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/). This code is based on a study conducted by the [TODO Group](https://todogroup.org/) of many codes used in the open source community. If you have feedback about this code, contact our Response Team at the address listed above.

Contributing.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# How to contribute
2+
First, thanks for taking the time to contribute to our project! The following information provides a guide for making contributions.
3+
4+
## Code of Conduct
5+
6+
By participating in this project, you agree to abide by the [Oath Code of Conduct](Code-of-Conduct.md). Everyone is welcome to submit a pull request or open an issue to improve the documentation, add improvements, or report bugs.
7+
8+
## How to Ask a Question
9+
10+
If you simply have a question that needs an answer, [create an issue](https://help.github.com/articles/creating-an-issue/), and label it as a question.
11+
12+
## How To Contribute
13+
14+
### Report a Bug or Request a Feature
15+
16+
If you encounter any bugs while using this software, or want to request a new feature or enhancement, feel free to [create an issue](https://help.github.com/articles/creating-an-issue/) to report it, make sure you add a label to indicate what type of issue it is.
17+
18+
### Contribute Code
19+
Pull requests are welcome for bug fixes. If you want to implement something new, please [request a feature first](#report-a-bug-or-request-a-feature) so we can discuss it.
20+
21+
#### Creating a Pull Request
22+
Before you submit any code, we need you to agree to our [Contributor License Agreement](https://yahoocla.herokuapp.com/); this ensures we can continue to protect your contributions under an open source license well into the future.
23+
24+
Please follow [best practices](https://github.com/trein/dev-best-practices/wiki/Git-Commit-Best-Practices) for creating git commits.
25+
26+
When your code is ready to be submitted, you can [submit a pull request](https://help.github.com/articles/creating-a-pull-request/) to begin the code review process.

README.md

Lines changed: 38 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,35 +3,40 @@ Copyright 2017 Yahoo Inc.
33
Licensed under the terms of the Apache 2.0 license.
44
Please see LICENSE file in the project root for terms.
55
-->
6-
[![Build Status](https://travis-ci.org/yahoo/TensorFlowOnSpark.svg?branch=master)](https://travis-ci.org/yahoo/TensorFlowOnSpark) [![PyPI version](https://badge.fury.io/py/tensorflowonspark.svg)](https://badge.fury.io/py/tensorflowonspark)
7-
86
# TensorFlowOnSpark
7+
> _TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark
8+
clusters._
99

10-
## What's TensorFlowOnSpark?
10+
[![Build Status](https://travis-ci.org/yahoo/TensorFlowOnSpark.svg?branch=master)](https://travis-ci.org/yahoo/TensorFlowOnSpark) [![PyPI version](https://badge.fury.io/py/tensorflowonspark.svg)](https://badge.fury.io/py/tensorflowonspark)
1111

12-
TensorFlowOnSpark brings scalable deep learning to [Apache Hadoop](http://hadoop.apache.org) and [Apache Spark](http://spark.apache.org)
13-
clusters. By combining salient features from deep learning framework
14-
[TensorFlow](https://www.tensorflow.org) and big-data frameworks
15-
Apache Spark and Apache Hadoop, TensorFlowOnSpark enables distributed
12+
By combining salient features from the [TensorFlow](https://www.tensorflow.org) deep learning framework with [Apache Spark](http://spark.apache.org) and [Apache Hadoop](http://hadoop.apache.org), TensorFlowOnSpark enables distributed
1613
deep learning on a cluster of GPU and CPU servers.
1714

18-
TensorFlowOnSpark enables distributed TensorFlow training and
19-
inference on Apache Spark clusters. It seeks to minimize the amount
15+
It enables both distributed TensorFlow training and
16+
inferencing on Spark clusters, with a goal to minimize the amount
2017
of code changes required to run existing TensorFlow programs on a
2118
shared grid. Its Spark-compatible API helps manage the TensorFlow
2219
cluster with the following steps:
2320

2421
1. **Startup** - launches the Tensorflow main function on the executors, along with listeners for data/control messages.
2522
1. **Data ingestion**
26-
- **Readers & QueueRunners** - leverages TensorFlow's [Reader](https://www.tensorflow.org/how_tos/reading_data/#reading_from_files) mechanism to read data files directly from HDFS.
27-
- **Feeding** - sends Spark RDD data into the TensorFlow nodes using the [feed_dict](https://www.tensorflow.org/how_tos/reading_data/#feeding) mechanism. Note that we leverage the [Hadoop Input/Output Format](https://github.com/tensorflow/ecosystem/tree/master/hadoop) for access to TFRecords on HDFS.
23+
- **InputMode.TENSORFLOW** - leverages TensorFlow's built-in APIs to read data files directly from HDFS.
24+
- **InputMode.SPARK** - sends Spark RDD data to the TensorFlow nodes via the [feed_dict](https://www.tensorflow.org/how_tos/reading_data/#feeding) mechanism. Note that we leverage the [Hadoop Input/Output Format](https://github.com/tensorflow/ecosystem/tree/master/hadoop) to access TFRecords on HDFS.
2825
1. **Shutdown** - shuts down the Tensorflow workers and PS nodes on the executors.
2926

30-
TensorFlowOnSpark was developed by Yahoo for large-scale distributed
31-
deep learning on our Hadoop clusters in Yahoo's private cloud.
27+
## Table of Contents
28+
29+
- [Background](#background)
30+
- [Install](#install)
31+
- [Usage](#usage)
32+
- [API](#api)
33+
- [Contribute](#contribute)
34+
- [License](#license)
3235

36+
## Background
3337

34-
## Why TensorFlowOnSpark?
38+
TensorFlowOnSpark was developed by Yahoo for large-scale distributed
39+
deep learning on our Hadoop clusters in Yahoo's private cloud.
3540

3641
TensorFlowOnSpark provides some important benefits (see [our
3742
blog](http://yahoohadoop.tumblr.com/post/157196317141/open-sourcing-tensorflowonspark-distributed-deep))
@@ -43,19 +48,29 @@ over alternative deep learning solutions.
4348
* Easily integrate with your existing data processing pipelines and machine learning algorithms (ex. MLlib, CaffeOnSpark);
4449
* Easily deployed on cloud or on-premise: CPU & GPU, Ethernet and Infiniband.
4550

51+
## Install
52+
53+
TensorFlowOnSpark is provided as a pip package, which can be installed on single machines via:
54+
```
55+
pip install tensorflowonspark
56+
```
57+
58+
For distributed clusters, please see our [wiki site](../../wiki) for detailed documentation for specific environments, such as our getting started guides for [single-node Spark Standalone](https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_Standalone), [YARN clusters](../../wiki/GetStarted_YARN) and [AWS EC2](../../wiki/GetStarted_EC2).
59+
60+
## Usage
61+
62+
To use TensorFlowOnSpark with an existing TensorFlow application, you can follow our [Conversion Guide](../../wiki/Conversion-Guide) to describe the required changes. Additionally, our [wiki site](../../wiki) has pointers to some presentations which provide an overview of the platform.
63+
64+
## API
65+
66+
[API Documentation](https://yahoo.github.io/TensorFlowOnSpark/) is automatically generated from the code.
4667

47-
## Using TensorFlowOnSpark
68+
## Contribute
4869

49-
Please check TensorFlowOnSpark [wiki site](../../wiki) for detailed
50-
documentations such as getting started guides for [YARN
51-
cluster](../../wiki/GetStarted_YARN) and [AWS EC2
52-
cluster](../../wiki/GetStarted_EC2). A [Conversion
53-
Guide](../../wiki/Conversion-Guide) has been provided to help you convert
54-
your TensorFlow programs.
70+
Please join the [TensorFlowOnSpark user group](https://groups.google.com/forum/#!forum/TensorFlowOnSpark-users) for discussions and questions. If you have a question, please review our [FAQ](../../wiki/Frequently-Asked-Questions) before posting.
5571

56-
## Mailing List
72+
Contributions are always welcome. For more information, please see our [guide for getting involved](Contributing.md).
5773

58-
Please join [TensorFlowOnSpark user group](https://groups.google.com/forum/#!forum/TensorFlowOnSpark-users) for discussions and questions.
5974

6075
## License
6176

0 commit comments

Comments
 (0)