Provisioning vs deployment

Why isn't all the server configuration in Puppet?

Provisioning

Gives base infrastructure and maintains state

Tools

• Puppet
  • declarative
    • declare a state that you want the machine to be in, and puppet 'makes it so'
  • configured to run every 20 minutes

Deployment

Human is explicitly changing the state of the deployed service from one to another

Requirements

1. co-ordinated deployment (you don't want each web node to deploy randomly in the next 20minutes, when it could cause breakage)
2. rollback
3. doable by people without root (puppet changes can only be merged by ops)
4. separate from the provisioning. if puppet breaks, it doesn't immediately cause downtime, while if a deploy breaks, it does.

Deployment Tools

• Scap3
• Fabric
• Trebuchet (deprecated)

Examples that illustrate the difference

Deploying a git repo

• Provisioner does the initial repo checkout in order to setup the appropriate structure and permissions
• Deployer pulls into the repo and restarts services without the need for root (ideally)

IRC conversation that's not captured above

schana: I think I'm missing why you wouldn't have all the configuration in puppet, and have the fabfile simply kick the agent on all the relevant hosts

yuvipanda: so in an *ideal* world, we can do continuous deploys via puppet or similar service, that constantly just deploys master.
yuvipanda: without any human intervention.

schana: so you don't use puppet in a way that sets up logic between items (or whatever the proper puppet term is) - i.e. you wouldn't have a service restart when a file changes?

yuvipanda: you would, but only for 'base infrastructure'
yuvipanda: for example, if /etc/ldap.yaml changes, our base puppet is setup to restart nscd and other services that read that.
yuvipanda: but it isn't used for software we write and deploy ourselves.
yuvipanda: and even there, puppet will only know the file has changed if it was changed by puppet itself
yuvipanda: and only on the next run
yuvipanda: puppet can also be broken by unrelated things
yuvipanda: for example, a package gets added to base (which is included on all machines)
yuvipanda: that can break puppet if done wrongly
yuvipanda: and we don't want to couple deployment breaking from that
yuvipanda: but primarily, it's 1 - 4 i said earlier
yuvipanda: and puppet sucks at all of those things
yuvipanda: there's no form of 'rollback' that isn't a massive unreliable hack, for example
yuvipanda: (3) is a deal breaker since only a grand total of 24 people have +2 on ops/puppet repository
yuvipanda: 1-3 is primary reason why provisioning (setting up infrastructure) is separate from deployment (setting up and changing state of service)
yuvipanda: if you also do a 'git log modules/ores' that will have far fewer commits than the deployment repo
yuvipanda: since once you've it setup, you kinda get to take infrastructure for granted and not need to modify it that much
yuvipanda: the uwsgi parameters are in puppet, for example.
yuvipanda: but I agree the split is confusing.
yuvipanda: back when I started doing this, I was like 'oh I know, I will just use git::clone to autodeploy!'
yuvipanda: and then it bites you hard in many many places
yuvipanda: and then you go 'ok, let me not do that next time'
yuvipanda: part of the problem is that most people when they get to dealing with deployment systems are primarily dealing with mediawiki's, which is so complex that it is kinda obvious that puppet, with its declarativeness and lack-of-good-ordering is not a good fit.
yuvipanda: but it's harder to see when you're deploying simpler services.
yuvipanda: for example, there's no good way in puppet to say 'run pip install if requirements.txt file changes'
yuvipanda: while you can do that easily in fab
yuvipanda: puppet only knows about changes that it does itself, doesn't know about changes done by other things (like a 'git clone' or a 'git pull' for example)