Assignment 2
STILL A DRAFT BUT SLIGHTLY BETTER NOW
TNI has hired you to finish their half-completed project: My Little Image Sharer (you guessed it: an image sharing app). The previous developers were terrible and fired; so you've been left with server and worker programs which receive uploaded images from users and resize images respectively. The server has been mostly completed, being able to put images into cloud storage while the worker has only the basic image processing functionality done. In order to make the app usable by TNI's users, you must communicate the arrival of a new image from the server to the worker by putting a message in the work queue, and then listening for that message in the worker; when the worker receives said message, it needs to get the existing image that was uploaded by the server, scale it to the appropriate thumbnail sizes, and then upload the thumbnails back to cloud storage.
DO NOT START THIS PART BEFORE FINISHING THE FIRST PART OR YOUR LIFE WILL BE FILLED WITH REGRET!!!! Unfortunately, simply completing the code for My Little Image Sharer won't be enough to save it from failing when hordes of emotionally-underdeveloped cloud system developers suddenly upload or view all the images of love, tolerance and distributed systems they've been waiting to share with the world. TNI needs to have an app that scales and there are (at least) two scaling issues present: the worker system could become under-provisioned (too many images, not enough resizing power) and the data delivery network could become under-provisioned (too many requests, not enough network bandwidth). Ensure that your system can avoid these under-provisioning scenarios by automatically spinning up more workers when they get overwhelmed and by having your images distributed over a CDN for viewing.
Even after you've gotten through the blood, sweat and tears, TNI being the horribly managed company it is, wants a brief overview of what your app promises to deliver and what it doesn't so the next people they hire can continue developing My Little Image Sharer.
- server.py
- Generate an id for new images (Print id to console from Python)
- Put message into SQS with the new id (Check AWS console for messages)
- View image from the generated url (Download image with curl)
- worker.py
- Read messages from of SQS (Print messages to console from Python)
- Delete messages from SQS (Check AWS console to see messages are being removed)
- Save resized images back into S3 (Check AWS console and view items in your bucket)
- CloudWatch
- The worker.py script must start at boot. (Reboot your instance, "ps aux | grep worker.py")
- You must have an AMI with your worker embedded in it. (Spin up new instance manually with your AMI)
- Ensure a second instance spins up when first one becomes overloaded (Use "stress" to create CPU spikes on the first instance and then monitor both your EC2 instances and CloudWatch metrics in the AWS console)
- CloudFront
- Be able to download data from your S3 images bucket over CloudFront (Test generated CloudFront URLs)
- Modify server.py to return CloudFront URLs (Download image with curl)
-
Students must:
-
Work in groups of size 0 < n < 5
-
Stack:
-
Amazon EC2 (Boto)
-
Amazon S3 (Boto)
-
Amazon SQS (Boto)
-
Amazon CloudWatch
-
Amazon CloudFront
-
Python
-
Boto
-
Pillow
-
curl
-
Image sizes must be:
-
Small: 100x100
-
Medium: 300x300
-
Large: 600x600
-
Original: Unspecified
-
POST / must:
-
respond 400 Bad Request on non-existant or invalid image
-
respond 405 Method Not Allowed on anything other than POST
-
respond 406 Not Acceptable for anything other than Accept: application/json
-
respond 413 Request Entity Too Large for images over 1MB
-
respond 202 Accepted for a valid image file named "image"
-
return with Content-Type "application/json"
-
return a JSON object of the format { original: "url...", small: "url...", ... }
-
GET "url..." must:
-
respond with 200 OK (competency +1)
-
respond with the correct Content-Type header (e.g. image/jpeg)
-
respond with an image file of the correct size (small/medium/large/original)
- How are these terms relevant or related to your app:
- data center
- virtual machine
- virtualization
- provisioning
- overprovisioned
- oversubscribed
- elastic computing
- utilization
- throughput
- latency
- API
- What does the Amazon SLA mean you can expect from your app?
- How does or doesn't your ID generation algorithm prevent conflicts?
- What platform-level, cluster-level, and application-level software is being used in your app?
- How does your app scale?
- How might it fail? (Hint: server.py)
- How could you change it to scale better?
- What other existing apps might use a similar platform? Why? (Hint: Video.)
- What metric did you chose for your AutoScaler/CloudWatch alarm? Why?
- If a worker fails while encoding an image, what happens? Can your system recover?
- Focus on core requirements first
- Use "curl" to write simple tests
- Start today
- Ask lots of questions
- Pick a requirement to satisfy
- Write a test that validates the requirement
- Implement the code to make the test succeed
- Upload Python code
- Worker Python code
- URL to your API
- PDF/txt/md/HTML answers to questions
You can post this information anywhere you'd like.
- 50% competency - demonstrate your understanding of core concepts
- 20% understanding - articulate both the breadth and depth of your knowledge
- 10% dedication - show a level of not inconsiderate effort
- 10% innovation - solve problems with an uncommon degree of creativity
- 10% technical - provide solutions that care about the details others overlook
You must complete all competency requirements to acheive a pass. Other requirements are additive to their maximum values.
- Technical Mastery: Complete all technical requirements