diff --git a/.DS_Store b/.DS_Store new file mode 100755 index 0000000..e157a5a Binary files /dev/null and b/.DS_Store differ diff --git a/Distributed Storage.docx b/Distributed Storage.docx new file mode 100644 index 0000000..ffcb08a Binary files /dev/null and b/Distributed Storage.docx differ diff --git a/Image/.DS_Store b/Image/.DS_Store new file mode 100755 index 0000000..5008ddf Binary files /dev/null and b/Image/.DS_Store differ diff --git a/Image/ML Real-Time Model.png b/Image/ML Real-Time Model.png new file mode 100644 index 0000000..16dc338 Binary files /dev/null and b/Image/ML Real-Time Model.png differ diff --git a/Image/MLmodel prediction.png b/Image/MLmodel prediction.png new file mode 100644 index 0000000..3d03539 Binary files /dev/null and b/Image/MLmodel prediction.png differ diff --git a/Image/Serverless app 1.png b/Image/Serverless app 1.png new file mode 100644 index 0000000..ca49328 Binary files /dev/null and b/Image/Serverless app 1.png differ diff --git a/Image/WX20181029-194711@2x.png b/Image/WX20181029-194711@2x.png new file mode 100755 index 0000000..fdd4fe9 Binary files /dev/null and b/Image/WX20181029-194711@2x.png differ diff --git a/Image/WX20181029-195157@2x.png b/Image/WX20181029-195157@2x.png new file mode 100755 index 0000000..d450b03 Binary files /dev/null and b/Image/WX20181029-195157@2x.png differ diff --git a/Image/WechatIMG1.png b/Image/WechatIMG1.png new file mode 100755 index 0000000..860cee8 Binary files /dev/null and b/Image/WechatIMG1.png differ diff --git a/Image/WechatIMG2.png b/Image/WechatIMG2.png new file mode 100755 index 0000000..3d074d6 Binary files /dev/null and b/Image/WechatIMG2.png differ diff --git a/Image/WechatIMG3.png b/Image/WechatIMG3.png new file mode 100755 index 0000000..b2c8664 Binary files /dev/null and b/Image/WechatIMG3.png differ diff --git a/Image/WechatIMG4.png b/Image/WechatIMG4.png new file mode 100755 index 0000000..5bd6126 Binary files /dev/null and b/Image/WechatIMG4.png differ diff --git a/Image/aws1.png b/Image/aws1.png new file mode 100755 index 0000000..23687c2 Binary files /dev/null and b/Image/aws1.png differ diff --git a/Image/aws10.png b/Image/aws10.png new file mode 100755 index 0000000..62c8ec7 Binary files /dev/null and b/Image/aws10.png differ diff --git a/Image/aws11.png b/Image/aws11.png new file mode 100755 index 0000000..bba0c33 Binary files /dev/null and b/Image/aws11.png differ diff --git a/Image/aws2.png b/Image/aws2.png new file mode 100755 index 0000000..c219062 Binary files /dev/null and b/Image/aws2.png differ diff --git a/Image/aws3.png b/Image/aws3.png new file mode 100755 index 0000000..1c83a91 Binary files /dev/null and b/Image/aws3.png differ diff --git a/Image/aws4.png b/Image/aws4.png new file mode 100755 index 0000000..9e9f7f6 Binary files /dev/null and b/Image/aws4.png differ diff --git a/Image/aws5.png b/Image/aws5.png new file mode 100755 index 0000000..728f000 Binary files /dev/null and b/Image/aws5.png differ diff --git a/Image/aws6.png b/Image/aws6.png new file mode 100755 index 0000000..ac13e20 Binary files /dev/null and b/Image/aws6.png differ diff --git a/Image/aws8.png b/Image/aws8.png new file mode 100755 index 0000000..549cc4b Binary files /dev/null and b/Image/aws8.png differ diff --git a/Image/aws9.png b/Image/aws9.png new file mode 100755 index 0000000..584d333 Binary files /dev/null and b/Image/aws9.png differ diff --git a/Image/bucket1.png b/Image/bucket1.png new file mode 100644 index 0000000..2b89d3e Binary files /dev/null and b/Image/bucket1.png differ diff --git a/Image/bucket2.png b/Image/bucket2.png new file mode 100644 index 0000000..07334fc Binary files /dev/null and b/Image/bucket2.png differ diff --git a/Image/bucket3.png b/Image/bucket3.png new file mode 100644 index 0000000..33a36cf Binary files /dev/null and b/Image/bucket3.png differ diff --git a/Image/bucket4.png b/Image/bucket4.png new file mode 100644 index 0000000..eb2edf2 Binary files /dev/null and b/Image/bucket4.png differ diff --git a/Image/bucket5.png b/Image/bucket5.png new file mode 100644 index 0000000..3caf38c Binary files /dev/null and b/Image/bucket5.png differ diff --git a/Image/bucket6.png b/Image/bucket6.png new file mode 100644 index 0000000..5b9f0ea Binary files /dev/null and b/Image/bucket6.png differ diff --git a/Image/cloud1.png b/Image/cloud1.png new file mode 100755 index 0000000..c7a4d53 Binary files /dev/null and b/Image/cloud1.png differ diff --git a/Image/cloud2.png b/Image/cloud2.png new file mode 100755 index 0000000..1781d1e Binary files /dev/null and b/Image/cloud2.png differ diff --git a/Image/cloud3.png b/Image/cloud3.png new file mode 100755 index 0000000..b7dcd14 Binary files /dev/null and b/Image/cloud3.png differ diff --git a/Image/cloud4.png b/Image/cloud4.png new file mode 100644 index 0000000..0b6564c Binary files /dev/null and b/Image/cloud4.png differ diff --git a/Image/cloud5.png b/Image/cloud5.png new file mode 100644 index 0000000..f19db3b Binary files /dev/null and b/Image/cloud5.png differ diff --git a/Image/dn 1.png b/Image/dn 1.png new file mode 100755 index 0000000..10637c5 Binary files /dev/null and b/Image/dn 1.png differ diff --git a/Image/dn10.png b/Image/dn10.png new file mode 100755 index 0000000..ef3179e Binary files /dev/null and b/Image/dn10.png differ diff --git a/Image/dn11.png b/Image/dn11.png new file mode 100755 index 0000000..8583fde Binary files /dev/null and b/Image/dn11.png differ diff --git a/Image/dn12.png b/Image/dn12.png new file mode 100755 index 0000000..4863f3f Binary files /dev/null and b/Image/dn12.png differ diff --git a/Image/dn13.png b/Image/dn13.png new file mode 100755 index 0000000..590fb2d Binary files /dev/null and b/Image/dn13.png differ diff --git a/Image/dn14.png b/Image/dn14.png new file mode 100755 index 0000000..6d0bbe7 Binary files /dev/null and b/Image/dn14.png differ diff --git a/Image/dn2.png b/Image/dn2.png new file mode 100755 index 0000000..9492073 Binary files /dev/null and b/Image/dn2.png differ diff --git a/Image/dn3.png b/Image/dn3.png new file mode 100755 index 0000000..d89c820 Binary files /dev/null and b/Image/dn3.png differ diff --git a/Image/dn4.png b/Image/dn4.png new file mode 100755 index 0000000..59ad121 Binary files /dev/null and b/Image/dn4.png differ diff --git a/Image/dn5.png b/Image/dn5.png new file mode 100755 index 0000000..19e62c2 Binary files /dev/null and b/Image/dn5.png differ diff --git a/Image/dn6.png b/Image/dn6.png new file mode 100755 index 0000000..51dde8d Binary files /dev/null and b/Image/dn6.png differ diff --git a/Image/dn7.png b/Image/dn7.png new file mode 100755 index 0000000..7baf4c1 Binary files /dev/null and b/Image/dn7.png differ diff --git a/Image/dn8.png b/Image/dn8.png new file mode 100755 index 0000000..a544b3c Binary files /dev/null and b/Image/dn8.png differ diff --git a/Image/dn9.png b/Image/dn9.png new file mode 100755 index 0000000..cf8689d Binary files /dev/null and b/Image/dn9.png differ diff --git a/Image/docker t3 - 2.png b/Image/docker t3 - 2.png new file mode 100755 index 0000000..53da4d6 Binary files /dev/null and b/Image/docker t3 - 2.png differ diff --git a/Image/docker t3-1.png b/Image/docker t3-1.png new file mode 100755 index 0000000..5562155 Binary files /dev/null and b/Image/docker t3-1.png differ diff --git a/Image/doing more 1.png b/Image/doing more 1.png new file mode 100755 index 0000000..68681fc Binary files /dev/null and b/Image/doing more 1.png differ diff --git a/Image/doing more 2.png b/Image/doing more 2.png new file mode 100755 index 0000000..a0bf2b9 Binary files /dev/null and b/Image/doing more 2.png differ diff --git a/Image/was7.png b/Image/was7.png new file mode 100755 index 0000000..7badd7b Binary files /dev/null and b/Image/was7.png differ diff --git a/README.md b/README.md deleted file mode 100644 index 37d2d46..0000000 --- a/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Distributed Systems Practice -Notes from learning about distributed systems in [GW CS 6421](https://gwdistsys18.github.io/) with [Prof. Wood](https://faculty.cs.gwu.edu/timwood/) - -## Area 1 -> Include notes here about each of the links - -## Area 2 -> Include notes here about each of the links diff --git a/Round 1 Note.md b/Round 1 Note.md new file mode 100755 index 0000000..4776211 --- /dev/null +++ b/Round 1 Note.md @@ -0,0 +1,190 @@ +# video:why docker? (20min) + +Docker is an open source object, release in 2013. + +The most great thing that docker has is higher speed. We can do faster develop by using docker. + +But, docker is not good at deal difficult problem, like multitype application. + +---- +# Lab: DevOps Docker Beginners Guide (80min) + +I learned some basic operating about docker. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/WechatIMG1.png?raw=true) + +Images - The file system and configuration of our application which are used to create containers. + +Containers - Running instances of Docker images. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/WechatIMG2.png?raw=true) + +Alpine - lightweight Linux distribution so it is quick to pull down and run, making it a popular starting point for many other images. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/WechatIMG3.png?raw=true) + +docker container start "container ID": pattern. + +---- + +# Video: What are Containers? (35min) + + Containers are isolated from each other and bundle their own tools, libraries and configuration files; they can communicate with each other through well-defined channels. All containers are run by a single operating system kernel and are thus more lightweight than virtual machines. Containers are created from "images" that specify their precise contents. Images are often created by combining and modifying standard images downloaded from public repositories. + +----- + +**Video: VMs Versus Containers** (20min) + +Both VMs and container is good choice for build your project. + +* container is lightweight, VMs is Heavyweight. +* VMs works between the phyiscal infrastructure and OS layer, container works between the OS and your app. + +- + +# Lab: Docker Intro （55min） + +### Task1: Run some simple Docker containers + +There are different ways to use containers. These include: + +* To run a single task: This could be a shell script or a custom app. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/WechatIMG4.png?raw=true) + +* Interactively: This connects you to the container similar to the way you SSH into a remote server. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/WX20181029-194711@2x.png?raw=true) + +* In the background: For long-running services like websites and databases. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/WX20181029-195157@2x.png?raw=true) + +### Task 2: Package and run a custom app using Docker + +Build a simple website image +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/docker%20t3-1.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/docker%20t3%20-%202.png?raw=true) + +### Task 3: Modify a running website + +* Start our web app with a bind mount + +* Modify the running website + +* Update the image + +* Push your images to Docker Hub + +---- + +# Lab: Doing more with Docker Images (100min) + +* Image creation from a container +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/doing%20more%201.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/doing%20more%202.png?raw=true) + +* Image creation using a Dockerfile + + *Dockerfile* - Docker can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. + +---- + +# Lab: Docker Networking (90min) + +### Section #1 - Networking Basics +* Step 1: The Docker Network Command +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn%201.png?raw=true) +* Step 2: List networks +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn2.png?raw=true) +* Step 3: Inspect a network +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn3.png?raw=true) +* Step 4: List network driver plugins + ![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn4.png?raw=true) + +### Section #2 - Bridge Networking +* Step 1: The Basics + ![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn5.png?raw=true) +* Step 2: Connect a container +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn6.png?raw=true) +* Step 3: Test network connectivity +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn7.png?raw=true) +* Step 4: Configure NAT for external connectivity +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn8.png?raw=true) + +### Section #3 - Overlay Networking +* Step 1: The Basics +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn9.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn10.png?raw=true) +* Step 2: Create an overlay network +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn11.png?raw=true) +Create a new overlay network won't display/created in second host. +* Step 3: Create a service + ![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn12.png?raw=true) +we created service here +"docker network inspect" only shows containers/tasks running on the local node. +* Step 4: Test the network +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn13.png?raw=true) +* Step 5: Test service discovery +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/dn14.png?raw=true) + +---- + +# Lab: Swarm Mode Introduction +* **Cluster management integrated with Docker Engine**: Use the Docker Engine CLI to create a swarm of Docker Engines where you can deploy application services. You don’t need additional orchestration software to create or manage a swarm. +* **Decentralized design**: Instead of handling differentiation between node roles at deployment time, the Docker Engine handles any specialization at runtime. You can deploy both kinds of nodes, managers and workers, using the Docker Engine. This means you can build an entire swarm from a single disk image. +* **Declarative service model**: Docker Engine uses a declarative approach to let you define the desired state of the various services in your application stack. For example, you might describe an application comprised of a web front end service with message queueing services and a database backend. +* **Scaling**: For each service, you can declare the number of tasks you want to run. When you scale up or down, the swarm manager automatically adapts by adding or removing tasks to maintain the desired state. +* **Desired state reconciliation**: The swarm manager node constantly monitors the cluster state and reconciles any differences between the actual state and your expressed desired state. For example, if you set up a service to run 10 replicas of a container, and a worker machine hosting two of those replicas crashes, the manager creates two new replicas to replace the replicas that crashed. The swarm manager assigns the new replicas to workers that are running and available. +* **Multi-host networking**: You can specify an overlay network for your services. The swarm manager automatically assigns addresses to the containers on the overlay network when it initializes or updates the application. +* **Service discovery**: Swarm manager nodes assign each service in the swarm a unique DNS name and load balances running containers. You can query every container running in the swarm through a DNS server embedded in the swarm. +* **Load balancing**: You can expose the ports for services to an external load balancer. Internally, the swarm lets you specify how to distribute service containers between nodes. +* **Secure by default**: Each node in the swarm enforces TLS mutual authentication and encryption to secure communications between itself and all other nodes. You have the option to use self-signed root certificates or certificates from a custom root CA. +* **Rolling updates**: At rollout time you can apply service updates to nodes incrementally. The swarm manager lets you control the delay between service deployment to different sets of nodes. If anything goes wrong, you can roll-back a task to a previous version of the service. +![](https://training.play-with-docker.com/images/ops-swarm-arch.svg) +![](https://training.play-with-docker.com/images/ops-swarm-stack-service-task.svg) + +- + +# Kubernetes (110min) +* Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. + +| Kubernets | Swarm | +|----------|:-------------:| +| Applications can be deployed using a combination of pods, deployments, and services (or “microservices”). A pod is a group of co-located containers and is the atomic unit of a deployment. A deployment can have replicas across multiple nodes. A service is the “external face” of container workloads and integrates with DNS to round-robin incoming requests. | Applications can be deployed as services (or “microservices”) in a Swarm cluster. Multi-container applications can specified using YAML files. Docker Compose can deploy the app. Tasks (an instance of a service running on a node) can be distributed across datacenters using labels. Multiple placement preferences can be used to distribute tasks further, for example, to a rack in a datacenter. | +| Each application tier is defined as a pod and can be scaled when managed by a deployment, which is specified in YAML. The scaling can be manual or automated. Pods can be used to run vertically integrated application stacks such as LAMP (Apache, MySQL, PHP) or ELK/Elastic (Elasticsearch, Logstash, Kibana), co-located and co-administered apps such as content management systems and apps for backups, checkpoint, compression, rotation, snapshotting. | Services can be scaled using Docker Compose YAML templates. Services can be global or replicated. Global services run on all nodes, replicated services run replicas (tasks) of the services across nodes. For example, A MySQL service with 3 replicas will run on a maximum of 3 nodes. Tasks can be scaled up or down, and deployed in parallel or in sequence. | +| Auto-scaling using a simple number-of-pods target is defined declaratively using deployments. CPU-utilization-per-pod target is available. Other targets are on the roadmap. | Not directly available. For each service, you can declare the number of tasks you want to run. When you manually scale up or down, the Swarm manager automatically adapts by adding or removing tasks. | +|The deployment controller supports both “rolling-update” and “recreate” strategies. Rolling updates can specify maximum number of pods unavailable or maximum number running during the process. | At rollout time, you can apply rolling updates to services. The Swarm manager lets you control the delay between service deployment to different sets of nodes, thereby updating only 1 task at a time.| +|With the release of 1.6, Kubernetes scales to 5,000-node clusters. Kubernetes scalability is benchmarked against the following Service Level Objectives (SLOs):|According to the Docker’s blog post on scaling Swarm clusters, Docker Swarm has been scaled and performance tested up to 30,000 containers and 1,000 nodes with 1 Swarm manager.| + +- +# AWS Tutorial: Break a Monolith Application into Microservices （forever... ） + +**record** +* 383544123906.dkr.ecr.us-east-1.amazonaws.com/guan + +1. Install application. - Docker, AWS CLI, Atom. + +2. Clone repo from https://github.com/awslabs/amazon-ecs-nodejs-microservices. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws1.png?raw=true) +3. Build and Push docker image. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws4.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws5.png?raw=true) +*remember '.' is very important!* +4. creat a ECS cluster. + ![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws2.png?raw=true) + ![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws3.png?raw=true) +5. checking cluster. + +*task table* +![task table](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws6.png?raw=true). +*instance table*. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/was7.png?raw=true) +6. Write a task Definition +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws8.png?raw=true) +7. Configure the Application Load Balancer: Target Group. + + VPC: vpc-06277877ad4593027 +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws9.png?raw=true) +8. Configure the Application Load Balancer: Listener +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws10.png?raw=true) +9. Deploy the Monolith as a Service + ![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/aws11.png?raw=true) +10. Break the Monolith. +11. Deploy Microservices +12. ![](https://d1.awsstatic.com/Developer%20Marketing/containers/5786%20Monolith%20Containers_6-switching-traffic.d54c5d1d16f4e86f2a712f862988fa05aa885b22.png) + diff --git a/Round2 Note Dec.9 .md b/Round2 Note Dec.9 .md new file mode 100644 index 0000000..7641bb2 --- /dev/null +++ b/Round2 Note Dec.9 .md @@ -0,0 +1,214 @@ +#Round 2 note + + +##QwikLab: Intro to S3 +Time: 55min + + +what is the S3? + + +Amazon Simple Storaeg Service - Amazon S3 is a cloud storage service. This lab will teach us how to use S3. + +Big data storage is a big challenge. Localize storage plan is high cost, time cosuming, and heavy. Amazon S3 is GOOD solution to this problem. + +Advantage: Safely, available when needed and scale as your want. + +###Start lab: qwiklab + +###Task1: Creat a Bucket +1. No uppercase. +2. Creat a bucket is very easy. Just use the guide. +3. Bucket is a storage service's implement. Use bucket as storage device(Visual). + + +###Task2: Upload an Object to the Bucket +1. Control panel is well-designed. Only need to click button. + +###Task3: Make your Object Public +1. My sheep link: https://s3-us-west-2.amazonaws.com/mybucket130616/sheep.png +2. You can manage the object public or private by S3 control panel. + +###Task4: Creat a bucket policy +1. Append your ARN with "/*". +2. This task is extend task of task 3. Here we use bucket policy to control resources. + +###Task5: Explore Versioning. +1. This task told us about how S3 control version. +2. Default setting in S3 display the lastest version. + + +##QwikLab: Intro to Amazon Redshift +Time: 55min + +What is Amazon Redshift? + +Amazon Redshift is a fast, scalable data warehouse. Compare to tranditional data warehouse, Redshift is simple, faster and cost-effictive. + +Difference for S3: It is the difference of database and data warehouse. Database(S3) is used to give data for specific service. Data warehouse(Redshift) store data for analysis. + +###Start Lab +###Task1: Launch an Amazon Redshift Cluster +1. Care about Quicklaunch and launch. +2. Multi-node and Single-node. +3. Launch a cluster need a lot of time!!! +4. Cluster configuration gives full information, like properties, node and status. + +###Task2: Launch Pgweb to Communicate with your Redshift Cluster +1. Pgweb is a web application which provide a friendly SQL interface to Redshift. +2. Pgweb can easy and directly communicate with Redshift. + +###Task3: Create a table +1. SQL code is simple. + +###Task4: Load Sample Data from Amazon S3 +1. Redshift can import data from S3. + +###Task5: Query Data +1. Learn some basic function about query,. +2. SELECT 'name' like that display the 'name'. +3. WHERE, FROM, ORDER BY, GROUP BY, NOT. + +*** + + +SELECT firstname, lastname,state + +FROM users + +WHERE liketheatre AND likeclassical AND state = 'Smith' + +ORDER BY firstname + +*** + +##Video: Short AWS Machine Learning Overview + +Time: 10min + +1. Three layers: + > Framework and Interfaces + + > ML Platforms: Amazon Sagemaker + + > Application Services + +2. Advantage: storage, compute, database, messaging, security base. +3. Machine Learning can improve results. + +##AWS Tutorial: Analyze Big Data with Hadoop + +Time: 70min + +What is Amazon EMR? + +A Managed cluster platform that simplifies running big data frameworks. + +Advantage of Amazon EMR: + +> Cost Savings +> +> AWS Integration +> +> Deployment +> +> Scalability and Flexibility +> +> Reliability +> +> Security +> +> Monitoring +> +> Management Interfaces + +1. Prepare an Amazon S3 bucket for store data. And this bucket needs 2 folder:'logs' and 'output'. +2. I have already got EC2 keypair in round 1. That is enough. +3. Launch an EMR cluster, use S3 and EC2 key. +4. Edit SSH access. +5. Here we use hive script. HiveQL is similiar with SQL, this is use to manipulate data warehouse and analytic data. +6. We load the Hive script and resource to analysis. Hive script is like a guide for hadoop to know how to do with data. + +##QwikLab: Intro to Amazon Machine Learning + +Time: 65min + +Amazon ML + +![Amazon ML Overview](https://us-west-2-aws-training.s3.amazonaws.com/awsu-spl/spl-35/images/overview.png) +In this lab, I will use Amazon ML model to predict if customer like this restaurant. +This lab will use S3 as datasource. + +Amazon ML is benifit from Amazon strong data and cloud compute techology. + +We use ML model which generate from datasource to predict future data. More data will lead higher rate of correct prediction. + +###Start Lab +###Task1: Upload Training Data +1. Download data file. +2. Upload data file to S3 bucket. Step is same as usual. + +###Task2: Create a Datasource +1. We use a new Amazon service - Amazon Machine Learning. +2. Link ML with S3(or Redshift). +3. Should give a target for Learning. + +###Task3: Creat a ML Model from the Datasource +1. In this task, we create a model from the restaurant.data(previous datasource). +2. We can create multiple models from single datasource. +3. It takes 8 minute. In common situation, it will spend 5 to 10 minutes. + +###Task4: Evaluate an ML Model. +1. If we want a correct result, it is important to correcct evaluate ML Model. +2. ML model will give us predict results. This prediction results will be compare with known results. +3. We can see rate of prediciton. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/MLmodel%20prediction.png?raw=true) + +###Task5: Generate Prediction from Your ML Model. +1. Two modes: + > batch mode + > real-time mode +2. With this mode, we can input new information and get a prediciton. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/ML%20Real-Time%20Model.png?raw=true) + +##AWS Tutorial: Build a Machine Learning Model +It is similiar with previous part(QwikLab: Intro to Amazon Machine Learning). Just use the different datasource. + +##Amazon SageMaker + +Time:75min + + +Amazon SageMaker is a fully managed machine learning service. SageMaker is clearly positioned to connect applications, especially unstructured video image, voice, and language text related applications. +Sagemaker is used to free data scientists from pre-processing. +Data scientists using sagemaker need only know a few models, and the use of interfaces. + + +##Bring it all together:Build a Serverless Real-Time Data Processing App + +###Overview +Serverless applications don’t require you to provision, scale, and manage any servers. You can build them for nearly any type of application or backend service, and everything required to run and scale your application with high availability is handled for you. + +> 1. Build a data stream +> 2. Aggregate data +> 3. Process streaming data +> 4. Store & query data + +![](https://d1.awsstatic.com/serverless/Lambda%20Resources%20images/Architecture_0.ca60f70f01876f7a4bc96686bc90adcf791039f7.png) + +1. We use new service: Amazon Cloud9. +2. AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. +3. First create an environmnet. Then we can use Amazon lambda. Looks like codeanywhere. +4. Cloud-based, it is important so I repeat it. +5. New service, Kinesis. Kinesis is work with data stream in real-time. +6. Kinesis can collect data stream. It uses as monitor or dectactor. +7. Cognito, Amazon service, is used to sychronized data. +8. IAM, Identity and Access Management. Use this to create a policy. +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/Serverless%20app%201.png?raw=true) +9. It is easy to run multi-thread. Just rename them. +10. Use Kinesis analytic application will get SQL-like application. +11. DynamoDB， is used to manage NoSQL Database. +12. Lambda. +13. Kinesis can dilivery stream to S3. So that S3 can use to store data. +14. Athena, query data in S3. +15. In bulid this app, we use a lot of Amazon service and link them together \ No newline at end of file diff --git a/Round2 note.md b/Round2 note.md new file mode 100644 index 0000000..14f1806 --- /dev/null +++ b/Round2 note.md @@ -0,0 +1,48 @@ +# Round2 Cloud Web Apps +---- + +## AWS Tutorial: Launch a VM (1hr) +Step 1: Launch an Amazon EC2 Instance +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/cloud1.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/cloud2.png?raw=true) +Launched instance +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/cloud3.png?raw=true) +*RECORD* Public IP:18.207.114.48 + +Step 2: Configure your Instance + +Step 3: Connect to your Instance +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/cloud4.png?raw=true) +Step 4: Terminate Your Instance + +meet problem with user name. +---- + +## QwikLab: Intro to S3 + +Be able to: + +* Create a bucket in Amazon S3 + +* Add an object to your bucket + +* Manage access permissions on an object + +* Create a Bucket Policy + +* Use bucket versioning + +Task1: Creat a Bucket +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/bucket1.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/bucket2.png?raw=true) +Task2: upload an Object to the Bucket + +Task3: Make Your Object Public +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/cloud5.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/bucket3.png?raw=true) +Task4: Creat a Bucket Policy +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/bucket4.png?raw=true) +![](https://github.com/GuanYuankai/dist-sys-practice/blob/master/Image/bucket6.png?raw=true) + +Task 5: Explore Versioning + diff --git a/~$stributed Storage.docx b/~$stributed Storage.docx new file mode 100644 index 0000000..ae5bd7a Binary files /dev/null and b/~$stributed Storage.docx differ