416 Distributed Systems: Assignment 4
Due: Feb 27 at 9PM
2016W2, Winter 2017
In this assignment you will use the Azure cloud to build a global distributed measurement platform to evaluate website performance and to detect regional content variation.
The Azure cloud is composed of several data-centers, located world-wide. Each data center hosts many thousands of machines that can be used (i.e., rented) for a price. In this assignment you will build a distributed measurement system and test it by deploying it on Azure cloud.
The purpose of your system is to use the Azure data-centers as different vantage points from which you can measure website performance and also regional website content variation. For example, your system will measure the latency to fetch the www.facebook.com page from around the world. It will also reveal whether or not the page content retrieved from these different locations varies. Your system will indicate whether or not there is regional specialization of website content. Although you will test and deploy your system on Azure, it will have nothing Azure-specific about it, e.g., it should be possible to run your system on the CS ugrad servers without any code modifications.
Your system will use a server process to coordinate several worker processes that run across the Azure cloud in different data centers. The server will receive jobs submissions from clients through an RPC-based API where a job is one specific website target. To service a client's job, the server must coordinate with worker processes, and then report the measurement results back to the client.
Using Azure: stop VMs when not using
We prepared a google slides presentation covering the basic workflow of getting a VM running on Azure for this/future assignments. To setup the Go environment in a VM you can use the azureinstall.sh script.
The default Azure subscription comes with a limitation of 20 cores per region. For this assignment you should not need cores above this limit. Though you can issue
A Piazza post covered some basics details of Azure. The most important of which is that each second that your VM is running it is draining your balance (yikes!). You should STOP your VMs when you are not using them. It's up to you to police your own account.
As in the previous two assignments we provide interfaces that your system must use to communicate with the outside world (RPC to the server and HTTP protocol to websites). Internal communication and coordination between the server and the worker processes is up to you to design and implement.
The server must listen to connections from one client as well as several workers. The server must be able to handle an arbitrary number of workers. The server must handle exactly one client that connects and invokes several RPCs. The client can invoke two kinds of RPCs against the server:
Latency to download the page from a worker: The MeasureWebsite RPC requires measuring latency between a website and a worker. A single latency measurement should be the total time (in milliseconds) that it takes the worker to retrieve the resource at the specified URI by the worker. This time will vary from one request to another. Therefore, the client specifies the samplesPerWorker as part of the RPC, which is the number of times a worker should perform the measurement. The per-worker statistics reported back to the client should be the min/median/max of the set of performed measurements.
RTT latency between worker and server: The GetWorkers RPC requires measuring latency between a worker and the server. This should be the round-trip time (in milliseconds) to ping-pong a message via UDP. As for measuring latency to download a page, the client specifies the samplesPerWorker as part of the RPC, and the per-worker statistics reported back to the client should be the min/median/max of the set of performed measurements.
You can test and debug your solution using the sample client code that we provide to you.
Assumptions you can make
Assumptions you cannot make
The assignment is extensible with two kinds of extra credit. You must create an EXTRACREDIT.txt file in your repository and specify in this file which extra credit features you have implemented.
Write two go programs called server.go and worker.go that behave according to the description above.
Server process command line usage:go run server.go [worker-incoming ip:port] [client-incoming ip:port]
Worker process command line usage:go run worker.go [server ip:port]
Download the client code. Please carefully read and follow the RPC data structure comments at top of file.
Rough grading rubric