From william at knowmad.com Thu Aug 26 04:17:25 2021 From: william at knowmad.com (William McKee) Date: Thu, 26 Aug 2021 07:17:25 -0400 Subject: [Charlotte-pm] Is there an Ansible solution to this deployment process? Message-ID: Hey Perl Mongers, Thanks for your input on helping me solve my deployment problem to a multi-tenant Dancer app. Here are the steps that I shared in the Zoom call last night: 5. Login to server as a user with sudo privs and cd to the sites directory (e.g., `cd /home/dancer/cv/sites/SITEURL`) 6. Setup the service to start/stop the service 1. `sudo cp bin/cv-SITENAME.service /etc/systemd/system/` 2. `sudo chmod 644 /etc/systemd/system/cv-SITENAME.service` 3. `sudo systemctl enable cv-SITENAME.service --now` 4. `sudo systemctl status cv-SITENAME.service` 7. Setup nginx 1. Copy the nginx.conf config file `sudo cp environments/SITENAME-nginx.conf /etc/nginx/sites-enabled` directory 2. Restart nginx (`sudo service nginx reload`) 8. Setup SSL 1. Run `sudo certbot --nginx` and choose the new domain to setup (use the non www domain) 2. Choose to redirect the HTTP traffic to HTTPS 3. *Do* run the configuration test at ssllabs.com (look for link in output from command) Is this an easy problem for Ansible? The `sudo` and certbot parts are where I see the challenges. For certbot, I've been using the "wizard" configurator in certbot and imagine there's a way to pass in all the settings that are needed on the command line. Running the test on ssllabs.com is something that I can do manually so long as I get back the link from Ansible output. Thanks for your suggestions! William -- William McKee Managing Partner at Knowmad Digital Marketing P 704.343.9330 Ext. 803 <704.343.9330>* ?*M (704) 737-6596?* ?*E william at knowmad.com* ?*W http://www.knowmad.com/ * ?* [image: facebook] [image: linkedin] [image: twitter] [image: instagram] Should we talk? Schedule a meeting with me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Thu Aug 26 06:29:00 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Thu, 26 Aug 2021 09:29:00 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: References: Message-ID: Hey William! Those steps are all easily Ansible, except for one that I have to think a little more. Instead of dumping a huge answer -- or one with the recipe and no explanation -- right here, I'll blurp it as steps during the day. I might be slow-ish as $dayjob is murder these days. :-) So, very first thing: documentation! Ansible actually comes with a cli command `ansible-doc` that is just as awesome as perldoc. It provides the manpages of the modules installed locally. And they come with examples and stuff. It's really nice. And then there is the main website: https://docs.ansible.com/ansible-core/devel/index.html And for those who like books, I have "Ansible: Up and Running" and recommend it. It's short and to the point, and served me well. Next step coming up shortly! `/anick On Thu, 26 Aug 2021, at 7:17 AM, William McKee wrote: > Hey Perl Mongers, > > Thanks for your input on helping me solve my deployment problem to a multi-tenant Dancer app. Here are the steps that I shared in the Zoom call last night: > >> 5. Login to server as a user with sudo privs and cd to the sites directory (e.g., `cd /home/dancer/cv/sites/SITEURL`) >> 6. Setup the service to start/stop the service >> 1. `sudo cp bin/cv-SITENAME.service /etc/systemd/system/` >> 2. `sudo chmod 644 /etc/systemd/system/cv-SITENAME.service` >> 3. `sudo systemctl enable cv-SITENAME.service --now` >> 4. `sudo systemctl status cv-SITENAME.service` >> 7. Setup nginx >> 1. Copy the nginx.conf config file `sudo cp environments/SITENAME-nginx.conf /etc/nginx/sites-enabled` directory >> 2. Restart nginx (`sudo service nginx reload`) >> 8. Setup SSL >> 1. Run `sudo certbot --nginx` and choose the new domain to setup (use the non www domain) >> 2. Choose to redirect the HTTP traffic to HTTPS >> 3. *Do* run the configuration test at ssllabs.com (look for link in output from command) >> > Is this an easy problem for Ansible? The `sudo` and certbot parts are where I see the challenges. > > For certbot, I've been using the "wizard" configurator in certbot and imagine there's a way to pass in all the settings that are needed on the command line. > Running the test on ssllabs.com is something that I can do manually so long as I get back the link from Ansible output. > > > Thanks for your suggestions! > William > > -- > > > > *William McKee* > Managing Partner at Knowmad Digital Marketing > > *P *704.343.9330 Ext. 803*??**M *(704) 737-6596? *??**E *william at knowmad.com*??**W *http://www.knowmad.com/ *??* > facebook > linkedin > twitter > instagram > > Should we talk? Schedule a meeting with me. > > > -- > Charlotte-pm mailing list > Charlotte-pm at pm.org > https://mail.pm.org/mailman/listinfo/charlotte-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Thu Aug 26 07:31:08 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Thu, 26 Aug 2021 10:31:08 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: References: Message-ID: <56918f43-82eb-4f08-a6b5-69869d5cc32e@www.fastmail.com> ## Step 1 of ? : What does the managed hosts need? ssh and Python. That's it. :-) Well, okay, maybe a tad more when you look more closely. You'll need a user account on that machine to which you can connect (duh!). Best way is to drop your ssh key on the remote host so you can connect without interaction. If some actions will require root permissions, that user will also need sudo rights (also duh!). By default Ansible uses the `sudo` mechanism, but there are other alternatives too (https://docs.ansible.com/ansible/latest/user_guide/become.html) Since my main use case is my own machines, `sudo` always did the trick for me. ## Step 2 of ?: Herding the remote hosts For your usecase, I'll assume that you want the full list of hosts you manage, plus you want a sub-group of them that are the ones where you want to do that first install. To do that you define an inventory file, which can be INI-flavored: ``` yanick.ottawa.ca ansible_ssh_host=192.168.0.1 ansible_ssh_user=yanick jason.charlotte.us ansible_ssh_host=192.168.0.2 ansible_ssh_private_key_file=/home/yanick/.vagrant.d/insecure_private_key william.charlotte.us ansible_ssh_host=... ansible_become_pass=hushhush [onboarding] yanick.ottawa.ca jason.charlotte.us ``` Or YAML: ``` all: hosts: 'william.charlotte.us': ansible_ssh_host: 192.168.0.103 children: onboarding: hosts: 'yanick.ottawa.ca': ansible_ssh_host: 192.168.0.101 ansible_ssh_user: yanick ``` and to tie in to the scary features of YAML, I *think* it'd be also be possible to define that YAML file as ``` all: hosts: 'william.charlotte.us': ansible_ssh_host: 192.168.0.103 'yanick.ottawa.ca': &yanick ansible_ssh_host: 192.168.0.101 ansible_ssh_user: y children: onboarding: hosts: 'yanick.ottawa.ca': *yanick ``` but I'm not sure. ## Step 3 of ?: reality check Of course, I'm winging this as I go along. Take all bits and statement with a grain of salt. :-) More to come, `/anick -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Thu Aug 26 07:58:40 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Thu, 26 Aug 2021 10:58:40 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: <56918f43-82eb-4f08-a6b5-69869d5cc32e@www.fastmail.com> References: <56918f43-82eb-4f08-a6b5-69869d5cc32e@www.fastmail.com> Message-ID: <49f09d27-a612-4d37-a543-7b9c83ea92c4@www.fastmail.com> ## Step 4 of ?: Creating the onboard role In Ansible you have the notion of playbooks and roles. They both use the same syntax, and the difference is more organizational than anything else. A role is a reusable set of actions. So, e.g., you can have roles to install apache, create a set of users, or -- in this case -- onboard a new machine. A playbook is a set of actions you're applying to a group of hosts. Some of those actions might end up being roles, natch. Roles typically have a structured set of subdirectories. I won't list them all here, just mention the ones that will be relevant for this specific discussion: tasks/ # the actions of that role. The main file is main.yml files/ # any asset you need for that role. Template files, # scripts to copy, etc vars/ # variables we might want to use for that role. Main file is # also main.yml defaults/ # ditto, but with lower priority than `vars`. Don't think # too much about it now. handlers/ # when some actions trigger changes, you can auto-restart # services and whatnots. This is where those "react on change" # events are dealt with. In this use-case, your onboard role does 3 things: setup a service, configure nginx, and setup SSL. So let's mirror that in `tasks/main.yml`: ``` # in ./roles/onboard/tasks/main.yml - name: install service include_tasks: ./install_service.yml - name: setup nginx include_tasks: ./setup_nginx.yml - name: setup ssl include_tasks: ./setup_ssl.yml ``` More to come, `/anick` -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Thu Aug 26 09:14:07 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Thu, 26 Aug 2021 12:14:07 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: <49f09d27-a612-4d37-a543-7b9c83ea92c4@www.fastmail.com> References: <56918f43-82eb-4f08-a6b5-69869d5cc32e@www.fastmail.com> <49f09d27-a612-4d37-a543-7b9c83ea92c4@www.fastmail.com> Message-ID: <41b5762b-fa3f-4cf1-b4ba-1d174086b367@www.fastmail.com> ## Step 5 of ?: install_service.yml Ansible by default uses the Jinja2 templating system (pretty similar to Mustache and Handlebars). In your instructions you have SITENAME that is a per-host variable, so I'll go back to my inventory file and augment the host configs with it: ``` all: hosts: 'william.charlotte.us': ansible_ssh_host: 192.168.0.103 sitename: william children: onboarding: hosts: 'yanick.ottawa.ca': sitename: yanick ansible_ssh_host: 192.168.0.101 ansible_ssh_user: yanick ``` I'll assume that `cv.service` is a template (that I'll rename `cv.service.jinja2`) that might have some tweaks for the different installs. So let's do this: ``` # in roles/onboard/tasks/install_service.yml - name: install service script # become means "become the super-user" become: yes template: src: ../files/cv.service.jinja2 dest: /etc/systemd/system/cv.service mode: '644' - name: ensure the service is running service: name: cv enabled: yes state: started ``` More to come, `/anick` -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Thu Aug 26 09:20:26 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Thu, 26 Aug 2021 12:20:26 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: <41b5762b-fa3f-4cf1-b4ba-1d174086b367@www.fastmail.com> References: <56918f43-82eb-4f08-a6b5-69869d5cc32e@www.fastmail.com> <49f09d27-a612-4d37-a543-7b9c83ea92c4@www.fastmail.com> <41b5762b-fa3f-4cf1-b4ba-1d174086b367@www.fastmail.com> Message-ID: <89c50ffa-fb1d-4263-97ef-e2ecd9ea3744@www.fastmail.com> ## Step 6 of ?: set up nginx For the nginx config, let's leverage the handlers so that we'll only reload nginx if we change the config. ``` # in roles/onboard/handlers/main.yml - name: restart nginx service: name=nginx state=restarted become: yes ``` and the task file then becomes: ``` # in roles/onboard/tasks/setup_nginx.yml - name: copy config copy: src: ../files/nginx.conf dest: /etc/nginx/sites-enabled/{{sitename}}-nginx.conf notify: restart nginx ``` More to come, `/anick` -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Thu Aug 26 09:32:19 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Thu, 26 Aug 2021 12:32:19 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: <89c50ffa-fb1d-4263-97ef-e2ecd9ea3744@www.fastmail.com> References: <56918f43-82eb-4f08-a6b5-69869d5cc32e@www.fastmail.com> <49f09d27-a612-4d37-a543-7b9c83ea92c4@www.fastmail.com> <41b5762b-fa3f-4cf1-b4ba-1d174086b367@www.fastmail.com> <89c50ffa-fb1d-4263-97ef-e2ecd9ea3744@www.fastmail.com> Message-ID: ## Step 7 of ?: set up certbot That's the more difficult or tricky bit. In your steps you are using the interactive move of certbot. Although Ansible as an 'expect' module (https://docs.ansible.com/ansible/latest/collections/ansible/builtin/expect_module.html), I would strongly recommend to try to find a way to perform that config without interactions. That makes things so much easier. As for outputting some stuff from the remote host, you can always capture the output of commands, and then print them out as an action: ``` - name: get the location of the tool 'waldo' shell: whereis waldo register: waldo_res - debug: var=waldo_res.stdout ``` ## Step 8 of 8: putting it in a playbook Finally time to run that sucker. We need a playbook that will execute the role 'onboard' on all the hosts belonging to the onboarding group. ``` # in ./playbooks/onboard.yml - name: all aboard! hosts: onboarding roles: onboard ``` And with that you should be able to do $ ansible-playbook ./playbooks/onboard.yml and have the role be applied to all the machines. You could also do a dry run and see the diffs of files that would be changed. $ ansible-playbook ./playbooks/onboard.yml --check --diff Or do a lot of other things, like further limit the playbook to only one host without changing the playbook itself: $ ansible-playbook ./playbooks/onboard.yml -l yanick.ottawa.ca And, in a nutshell, that's it. :-) Tadah, `/anick` -------------- next part -------------- An HTML attachment was scrubbed... URL: From william at knowmad.com Thu Aug 26 19:22:57 2021 From: william at knowmad.com (William McKee) Date: Thu, 26 Aug 2021 22:22:57 -0400 Subject: [Charlotte-pm] Is there an Ansible solution to this deployment process? In-Reply-To: References: Message-ID: Yanick, Thank you for the step-by-step guide to Ansible deployment. I'm starting to see the general outline of the solution, and I still need some clarifications. Here's where I am now: 1. I only have 1 server that I need to deploy sites onto. Don't expect this to change. Can I simplify the server setting along the following lines: ``` all: hosts: 'knowmad.com': ansible_ssh_host: 192.168.0.1 ansible_ssh_user: dancer ``` 1a. What file do I put this information into? Does it belong in `./playbooks/onboard.yml`? 1b. Since I'm not onboarding a server but rather using ansible to deploy a new site within a multi-tenant web app, I may rename that file to `./playbooks/deploy.yml`. Does that make sense? Or does this sound confusing? 2. For the site specific configurations, I already have a perl script that uses Template based config files to build the domain specific configuration files needed for a new site. Perhaps I could replace that with Ansible + jinja2 at a future date. I'd need to figure out how to pass in the domain specific info when deploying a new site. 3. It appears that I'll need to build a new set of ansible tasks for each domain. Yes? Or, is there a way to define placeholders that can be passed into a task via the ansible command line? 4. Do I run the ansible client on my workstation and let it connect to the server? Do I need to install any ansible components on the server or only on my workstation? or both? 5. In addition to the site-specific files, I also need to push up any edits to the Perl libs, etc. I'm currently using rsync to do so. Wondering if I may be better off using ansible to manage the entire deployment process to the server. Thanks for your guidance. There's probably more questions but right now I'll be happy to just get a basic script running that can push the files into place on the server and setup the systemctl commands. William -------------- next part -------------- An HTML attachment was scrubbed... URL: From yanick at babyl.ca Fri Aug 27 07:06:54 2021 From: yanick at babyl.ca (Yanick Champoux) Date: Fri, 27 Aug 2021 10:06:54 -0400 Subject: [Charlotte-pm] =?utf-8?q?Is_there_an_Ansible_solution_to_this_de?= =?utf-8?q?ployment_process=3F?= In-Reply-To: References: Message-ID: Hi William! On Thu, 26 Aug 2021, at 10:22 PM, William McKee wrote: > 1. I only have 1 server that I need to deploy sites onto. Don't expect this to change. Can I simplify the server setting along the following lines: > > ``` > all: > hosts: > 'knowmad.com': > ansible_ssh_host: 192.168.0.1 > ansible_ssh_user: dancer > ``` Oooh. I see. You are adding configs to your own server each time you add a new customer, rather than going and installing something on their server. I misunderstood that the first time around. :-) And yes, absolutely, you can totally have an inventory of a single host. > > 1a. What file do I put this information into? Does it belong in `./playbooks/onboard.yml`? It's in `inventory.yml` or `inventory.ini`, dependending on the format you're using. > > 1b. Since I'm not onboarding a server but rather using ansible to deploy a new site within a multi-tenant web app, I may rename that file to `./playbooks/deploy.yml`. Does that make sense? Or does this sound confusing? I'd say that "deploy" is maybe a tad generic. But hey, if it works for you, it works for you. Mostly if for the time being you have only one playbook. :-) And that goes for the roles as well. In my example I went Full-Metal Best Practices and created a role, with task files for the different subtasks. You could totally throw everything in a single playbook, considering how little there is to it. > > 2. For the site specific configurations, I already have a perl script that uses Template based config files to build the domain specific configuration files needed for a new site. Perhaps I could replace that with Ansible + jinja2 at a future date. I'd need to figure out how to pass in the domain specific info when deploying a new site. As a first pass you can use your perl script to create the config files and populate the `roles/*/files` directory. But if you have time, throw a quick glance at Jinja2. Honestly it's extremely close to Mustache / TT / Mason, so changing from one to the other is pretty painless (well, that's depending of how tortuous your templates are, I guess). > > 3. It appears that I'll need to build a new set of ansible tasks for each domain. Yes? Or, is there a way to define placeholders that can be passed into a task via the ansible command line? Yup. You can override/define variables from the command line, and you can pass variables to roles. - name; do the thing, for sitename include_tasks: ./do_the_thing.yml vars: sitename: '{{ weird_env_variable }}' And on the command line $ ansible-playbook -e weird_env_variable=potato .... (of course you could also have done `ansible-playbook -e sitename=potato ....` directly and skip the var mapping in the task) > > 4. Do I run the ansible client on my workstation and let it connect to the server? Do I need to install any ansible components on the server or only on my workstation? or both? You run Ansible from your workstation. All that the remote hosts need is ssh enabled and Python. > > 5. In addition to the site-specific files, I also need to push up any edits to the Perl libs, etc. I'm currently using rsync to do so. Wondering if I may be better off using ansible to manage the entire deployment process to the server. In addition of calling the shell commands yourself, there are a few Perl-centric modules for Ansible. Like https://docs.ansible.com/ansible/latest/collections/community/general/cpanm_module.html (that one is core) and https://github.com/swfz/ansible-plenv (that one is part of the community set of roles) Joy, `/anick -------------- next part -------------- An HTML attachment was scrubbed... URL: