Scaling Docker #1 – High Availability Docker DevOps with CoreOS, Etcd, Fleet, Flannel & Nginx



Let's build a high-availability production docker cluster!
So web development with Docker pretty much ROCKS - it enables you to build and scale world-class apps that used to only be doable for those working at companies like Google and Netflix.

The (not-really-bad) bad news is you have to start thinking like a world-class developer to make full use of it. To go from basic docker container deployment to a full-scale docker system, you'll need to implement a few things:

The Diagram: https://drive.google.com/file/d/0B74l6vUHf9HrMzNrYUlyMm40RWs/view?usp=sharing
The Code: https://github.com/willrstern/production-docker-ha-architecture

THIS VIDEO:
- Cluster computing - CoreOS is our solution here. CoreOS allows us to make an expandable cluster of computers so we can add more computers should our application need more power to scale. CoreOS also allows us to run services on any machine in the cluster and they can all communicate with each other.

NEXT VIDEO: https://youtu.be/top0MlcKSw4
- Scheduling: we shouldn't manually choose which app runs on which machine - this idea doesn't work at scale. Fleet will be our scheduler that decides where our apps/services run. If a CoreOS machine goes down, fleet will reschedule any services that it was running onto a new machine.

VIDEO 3: https://youtu.be/1zJ8FfC-gmU
- Service Registration/Service Discovery: Since we never know what ip address or port a service will be running on, we need to register that service so other services can find it and communicate with it. We'll use Etcd and Flannel for this

- Public DNS: Once the app is working, we need a fixed way for the outside world to access it. We'll use Nginx and Confd for this. Confd will watch Etcd for service registration and when services come/go, it will build a new configuration file for Nginx and reload Nginx.

This docker tutorial will help you understand how to go through every step of this process.

code source

  • Hey great series! Genuine question, do you consider this best practice above Kubernetes or Nomad?

    View Comment
  • Are the things you're showing in this video Digital Ocean specific? (eg Coreos configurations) How will this translate using another service like AWS?

    View Comment
  • how does coreos know the cluster size? and if i set the cluster size to 3 do i need a minimum of 2 members to make etcd work?

    View Comment
  • wow this is awesome!

    View Comment
  • Nice walkthrough, fleet + flannel seem interesting for orchestration, just wondering why you wouldn't stick with native docker swarm though, what advantages does fleet + flannel bring? Thanks again :-)

    View Comment
  • Thanks for consistently uploading high quality content! This past summer I watched your video "Watch this if you want to become a web developer". Since then I learned a ton of cool stuff, like react, redux, rethinkdb, node, webpack, etc. Now I'm in the process of scaling my web application :D.

    View Comment