How the **** I have I never heard of/used "tmux" on a Linux box? I mean seriously, they had me at Control + B --> D. For those like me who had never heard of "tmux" it is a NECESSITY if you ever run any long running (or even short running) scripts that require you to be connected to a SSH session.
Let's pretend that you are upgrading a very important application on your server and this upgrade script does thousands of instructions and database queries etc etc. Now let's pretend that you are 4 hours in and all of a sudden a squirrel nibbles the fiber optic cable connecting you to the interwebs (yes, this happened). All of a sudden all of the processes that are running have been terminated because your SSH session was disconnected. Now you have to pray that it either finished, somehow is still running, and didn't corrupt itself and you have to roll back to a snapshot (you did make a snapshot before you started right?).
Now let's pretend you knew about "tmux" and let's run though this situation again. You first SSH into your server and then you immediately issue the command:
tmux
Now you get a nifty colored bar at the bottom of your SSH screen. Mine is green, but it will probably depend on your terminal settings. You seem to get a standard prompt like you had before. That green bar is the only thing that really lets you know you are in tmux. Now you issue your script commands. 4 hours in the rascally squirrel gets his teeth into your cables. After flipping out and finally realizing what happened you then proceed to go to your mom's house for some nice good old fashioned cable internet and cookies.
From there you connect to your server (VPN + SSH) and then you issue the command (as the SAME user that you ran tmux from before):
tmux attach
Now you are brought right back where you were!!! HAPPINESS!!!
Useless example:
$ tmux
Now in tmux:
$ nano hello.txt
If you purposely want to disconnect hold down the CONTROL key and press "b" then press "d" to "detach"
Close your SSH connection completely or go to another computer and connect to your server. Now as the same user type:
$ tmux attach
Violoa! you now are editing hello.txt!
To exit from tmux you can just type "exit" at the prompt. Here is a nifty site dedicated to more tmux use cases like splitting screens etc: https://danielmiessler.com/study/tmux/
Happy tmux-ing!
Tuesday, March 21, 2017
Tuesday, March 07, 2017
Why I Hate/Love Free Support - I'm looking at you Piwik!
Who doesn't like free? Of course as the age old adage goes, you get what you pay for and Piwik is no different. Piwik is an analytical platform for log analysis and real time (semi) user tracking via web service calls. All in all the product is decent. We are running a local version of their software and performing log analysis only.
We (meaning me) ran into problems during a few different stages of implementation. I posted 3 different times to their forum with only crickets as a response. One topic was viewed 138 times and the others around 50ish. Not a lot, but hey someone's looking. There weren't even any responses from that one jerk who says, "I have no idea, man" or the other jerk that says "just RTFM" or the jerk that says "I'm not going to do your schoolwork for you" or the last jerk that is obviously from the java forums who says "if you are too stupid to figure it out I can't help you" (you know who you are!).
There is one guy, Matthieu Aubry (the creator of Piwik) who does make great efforts to answer questions, but he appears to be just one guy (or ARE YOU?!?!?!). I do not fault him in any way and as a developer who also supports users I understand the lack of time he has to do both.
Piwik does have paid support contracts that will solve the world's problems, but it is rather expensive. For the little guy you are SOL. Piwik has a stackoverflow section, but that too is sparsely answered. In the end, I spent way too many hours troubleshooting and trying to figure things out, but hey you get what you pay for.
We (meaning me) ran into problems during a few different stages of implementation. I posted 3 different times to their forum with only crickets as a response. One topic was viewed 138 times and the others around 50ish. Not a lot, but hey someone's looking. There weren't even any responses from that one jerk who says, "I have no idea, man" or the other jerk that says "just RTFM" or the jerk that says "I'm not going to do your schoolwork for you" or the last jerk that is obviously from the java forums who says "if you are too stupid to figure it out I can't help you" (you know who you are!).
There is one guy, Matthieu Aubry (the creator of Piwik) who does make great efforts to answer questions, but he appears to be just one guy (or ARE YOU?!?!?!). I do not fault him in any way and as a developer who also supports users I understand the lack of time he has to do both.
Piwik does have paid support contracts that will solve the world's problems, but it is rather expensive. For the little guy you are SOL. Piwik has a stackoverflow section, but that too is sparsely answered. In the end, I spent way too many hours troubleshooting and trying to figure things out, but hey you get what you pay for.
Wednesday, February 01, 2017
There's an Nginx on my Pi and that's how I like it!
The Raspberry Pi is just awesome. It is perhaps one of the best CHEAP well made computers out there, especially with the model 3 available now. However there is one HUGE problem... what the heck am I going to do with this thing once the it loses its shimmer and shine (genies divine - yea you know what I'm talking about parents)? There are so many cool projects out there that can be done like the beloved magic mirror project or perhaps one of most useless, but technically awesome BeetBox. What is a guy to do. I have two Pis (Pies?, Pii?, Pi's?) one is a 2 and one a 3. I'm reserving my 3 for Retro Pi (perhaps a future post on this), but my 2 is just hanging out collecting dust. Well I did what every devops kind of guy would do - run my own website on it!
This is the story of how the little engine that I thought couldn't actually could. Raspberry Pi 2 + Nginx 1.10.2 + PHP 7 + MariaDB 10.0.29 (screw you Oracle for corrupting everything you touch ::cough:: Java, Solaris, MySQL, Oracle licensing in the cloud ::cough:: ). Spoiler alert - on my small little site, this is REALLY fast. It will never be blade server fast, but response time is mostly < 1s on static pages. On a few php it was < 2s on fresh caches. 13ms for a static html page, 56ms for a 42k image, 248ms for a small php file.
First off make sure your Pi is up to date:
Now make sure your Pi firmware is up to date and reboot:
Once you are up and running let's get MariaDB and client installed:
Now secure it - PLEASE do this even if you are just futzing around. Get used to security all the time.
Now comes a tricky part. In order to install PHP 7 and the latest Nginx you will need to add the "stretch" repository. Edit the file:
Now run:
If all goes well let's now run the installer for Nginx:
On one particular site I can't find now, a person recommended changing the user/group to pi:pi for Nginx. It wasn't explained, but I'm guessing it was so Nginx does run as a privileged account (although you still need root to start it up with a low port 80/443).
My site has a top level www.domain.com as well as a few subdomains a.domain.com and b.domain.com. To set this up you need to add a config for each. First off is creating your domain configs. Go to:
Restart Nginx
Restart PHP FPM:
Now what doesn't help anyone with a custom web site is DHCP on your local network. To fix this, let's force the Pi to a static ip:
You read that correctly, I'm using my wireless connection (wlan0) for my web server MWAHAHAHAHAHA. It should be noted that most hosting providers will BLOCK port 80 and 443 from even touching your internal network. You may need to call them up to ask them to open it. In my case I had to call them, but as long as you have a minimum internet speed package they will be so kind as to open it up for you. Also you will need to forward traffic on port 80/443 from your router to your Pi (or other custom port if you desire). If you set a static ip on your Pi I recommend either setting that same device to receive that static ip on the router side or exclude that ip from being leased by another device. Call me Nostradamus, I've been predicting lease conflict problems for 500 years now.
Next up is something that came about due to cheapness, FREE SSL. What's that I say 'FREE SSL' that can't be. There is no way GoDaddy, Thawte or any of the other ridiculously priced SSLcrooks Certificate Authorities would allow that. Well they did - it is an awesome Linux Foundation Collaborative Project called Let's Encrypt: https://letsencrypt.org/. This is a new (relatively) Certificate Authority that works on the premise that you have to renew your certificate every 90 days. Sure that sucks, but when you save $50 a year, I can set up a small cron job that will automatically renew my certificate for me. There are limitations like the site must have a public DNS entry (sorry, local intranet domains need not apply), IP's that are public can not be signed, 90 day expiration, no wildcard certs. However with the wild card option, you can have up to 100 names on the same certificate (not really sure how that works) and you can also apply for multiple domains. They cap it at 20 domain requests a week (https://letsencrypt.org/docs/rate-limits/). This is really for Linux boxes, but you can get it to work on Windows servers with some extra work.
I followed this guide partially for the Let's Encrypt and Nginx stuff in case you are wondering:
http://www.htpcguides.com/secure-nginx-reverse-proxy-with-lets-encrypt-on-ubuntu-16-04-lts/
In the case of my Pi, I had to first install git so I could clone the repository (yes you can just download the zip and go from there, but where's the fun in that)?
Now go into your certbot directory to issue some commands to "install" certbot-auto which does the heavy lifting:
The first time this will ask you for your email address. Add a legit email in case you need to do something with your certs later (retrieve them etc). Notice that I have two domains with two "-d" options. If you want to secure both www.domain.com and just domain.com enter them this way. Now it will support either. You can issue individual certs if you like for subdomains in just the same manner like this:
Notice that I pointed the webroot to the actual web root of the configured site in Nginx? This is necessary because the certbot program puts a file in there when it goes to validate your web site for issuing the cert. Once this completes it will tell you where the certs/pem that you need are located. You can choose to leave them there (recommended) or move them. Keep in mind that if you move the certs, when 90 days rolls around and the cert is renewed automatically you will need to then move the cert to the new location either via a script you write or manually.
Now you need to configure Nginx to play nicely with the cert. Edit the site of your choosing:
Add the following to the "Server" section:
There is a lot of stuff in there, let's take it line by line.
listen 443 ssl; - This says respond to web requests on port 443 (default ssl port)
ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem; - this is the location of the PEM full certificate chain that Let's Encrypt generates for you.
ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem; - the location of the private key that Let's Encrypt generated for you. Necessary for SSL encryption to work
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - This limits what SSL protocols are served and accepted by the web server. As protocols get hacked this will need to get updated.
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH'; - this is the latest list that I found of secure encryption ciphers that WILL NOT work on older browser like IE 6.
ssl_prefer_server_ciphers on; - This tells clients that if you are not running my ciphers, you can't connect - LOSER
ssl_session_cache shared:SSL:10m; - honestly don't know what this does. Guessing performance caching for repeat clients.
ssl_dhparam /etc/ssl/certs/dhparam.pem; - This is the wizard that helps get the "A" SSL rating. By default if you don't include this I got a "B" rating from Qualys SSL Labs (https://ssllabs.com/). The reasoning is that Diffie Hellman algorithm is more secure when establishing connections. The unfortunate downside to this is that you need to generate the DH pem file. On my Pi 2 this took an hour, but I read in some places it took over 24 hours. It is random so it will take as long as it takes.
To generate that pem file:
You can choose a 2048 bit key which will go MUCH faster, but that is up to you. Now don't forget to set up a cron job (as root) to try to renew your SSL certs DAILY. They recommend that you try to renew it twice a day in case their servers are not available. There is no penalty for trying to renew a certificate and it doesn't count towards rate limiting.
In the file add an entry like this:
Note: after a successful update you will need to restart Nginx. You can just restart it during your maintenance window or just do a quick restart a few minutes after you try to renew your cert. I elected for a restart at 2am every day. Just add it to your root crontab and you are all set:
Restart Nginx again and test to make sure you can hit your page via SSL. Finger crossed. If you can, now check your rating on https://ssllabs.com. If you are bold let them post your results on their leader board. I'm not that bold. Right now I have an "A" rating and if any new fun SSL bugs come out, I'll have to update my cipher list. Welcome to DevOps!
This is the story of how the little engine that I thought couldn't actually could. Raspberry Pi 2 + Nginx 1.10.2 + PHP 7 + MariaDB 10.0.29 (screw you Oracle for corrupting everything you touch ::cough:: Java, Solaris, MySQL, Oracle licensing in the cloud ::cough:: ). Spoiler alert - on my small little site, this is REALLY fast. It will never be blade server fast, but response time is mostly < 1s on static pages. On a few php it was < 2s on fresh caches. 13ms for a static html page, 56ms for a 42k image, 248ms for a small php file.
First off make sure your Pi is up to date:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get upgrade
Now make sure your Pi firmware is up to date and reboot:
sudo rpi-update
sudo reboot
sudo reboot
Once you are up and running let's get MariaDB and client installed:
sudo apt-get install mariadb-server mariadb-client
Now secure it - PLEASE do this even if you are just futzing around. Get used to security all the time.
mysql_secure_installation
Now comes a tricky part. In order to install PHP 7 and the latest Nginx you will need to add the "stretch" repository. Edit the file:
/etc/apt/sources.list
Add the line:
deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
Make sure that line is just below the first line for Jesse. It should look something like this:
deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
Add the line:
deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
Make sure that line is just below the first line for Jesse. It should look something like this:
deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
Now run:
sudo apt-get update
sudo apt-get install -t stretch php7.0 php7.0-curl php7.0-gd php7.0-fpm php7.0-cli php7.0-opcache php7.0-mbstring php7.0-xml php7.0-zip
Note the "-t stretch". If you have multiple repositories this will force a specific one. Good practice to use anyway.
sudo apt-get install -t stretch php7.0 php7.0-curl php7.0-gd php7.0-fpm php7.0-cli php7.0-opcache php7.0-mbstring php7.0-xml php7.0-zip
Note the "-t stretch". If you have multiple repositories this will force a specific one. Good practice to use anyway.
If all goes well let's now run the installer for Nginx:
sudo apt-get install -t stretch nginx
On one particular site I can't find now, a person recommended changing the user/group to pi:pi for Nginx. It wasn't explained, but I'm guessing it was so Nginx does run as a privileged account (although you still need root to start it up with a low port 80/443).
sudo nano /etc/php/7.0/fpm/pool.d/www.conf
Change:
user = pi
group = pi
group = pi
My site has a top level www.domain.com as well as a few subdomains a.domain.com and b.domain.com. To set this up you need to add a config for each. First off is creating your domain configs. Go to:
/etc/nginx/sites-available/
Now add in your domain file:
sudo nano domain.com
After you add your goodness, now go to:
/etc/nginx/sites-enabled
Remove the entry for default and then add a link to your new one:
sudo ln -s ../etc/nginx/sites-available/domain.com
Remove the entry for default and then add a link to your new one:
sudo ln -s ../etc/nginx/sites-available/domain.com
Restart Nginx
sudo service nginx restart
Restart PHP FPM:
sudo service php7.0-fpm restart
Now what doesn't help anyone with a custom web site is DHCP on your local network. To fix this, let's force the Pi to a static ip:
sudo nano /etc/dhcpcd.conf
At the end of the file add your information for your static local ip (leave the '/24' in the first line):
interface wlan0
static ip_address=192.168.1.200/24
static routers=192.168.1.100
static domain_name_servers=192.168.1.100
static ip_address=192.168.1.200/24
static routers=192.168.1.100
static domain_name_servers=192.168.1.100
You read that correctly, I'm using my wireless connection (wlan0) for my web server MWAHAHAHAHAHA. It should be noted that most hosting providers will BLOCK port 80 and 443 from even touching your internal network. You may need to call them up to ask them to open it. In my case I had to call them, but as long as you have a minimum internet speed package they will be so kind as to open it up for you. Also you will need to forward traffic on port 80/443 from your router to your Pi (or other custom port if you desire). If you set a static ip on your Pi I recommend either setting that same device to receive that static ip on the router side or exclude that ip from being leased by another device. Call me Nostradamus, I've been predicting lease conflict problems for 500 years now.
Next up is something that came about due to cheapness, FREE SSL. What's that I say 'FREE SSL' that can't be. There is no way GoDaddy, Thawte or any of the other ridiculously priced SSL
I followed this guide partially for the Let's Encrypt and Nginx stuff in case you are wondering:
http://www.htpcguides.com/secure-nginx-reverse-proxy-with-lets-encrypt-on-ubuntu-16-04-lts/
In the case of my Pi, I had to first install git so I could clone the repository (yes you can just download the zip and go from there, but where's the fun in that)?
sudo apt-get install git -y
sudo git clone https://github.com/certbot/certbot /home/pi/certbot
sudo git clone https://github.com/certbot/certbot /home/pi/certbot
Now go into your certbot directory to issue some commands to "install" certbot-auto which does the heavy lifting:
cd /home/pi/certbot
./certbot-auto certonly -a webroot --webroot-path=/var/www/domain.com -d www.domain.com -d domain.com
./certbot-auto certonly -a webroot --webroot-path=/var/www/domain.com -d www.domain.com -d domain.com
The first time this will ask you for your email address. Add a legit email in case you need to do something with your certs later (retrieve them etc). Notice that I have two domains with two "-d" options. If you want to secure both www.domain.com and just domain.com enter them this way. Now it will support either. You can issue individual certs if you like for subdomains in just the same manner like this:
./certbot-auto certonly -a webroot --webroot-path=/var/www/a.domain.com -d a.domain.com
Notice that I pointed the webroot to the actual web root of the configured site in Nginx? This is necessary because the certbot program puts a file in there when it goes to validate your web site for issuing the cert. Once this completes it will tell you where the certs/pem that you need are located. You can choose to leave them there (recommended) or move them. Keep in mind that if you move the certs, when 90 days rolls around and the cert is renewed automatically you will need to then move the cert to the new location either via a script you write or manually.
Now you need to configure Nginx to play nicely with the cert. Edit the site of your choosing:
sudo nano /etc/nginx/sites-available/domain.com
Add the following to the "Server" section:
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
There is a lot of stuff in there, let's take it line by line.
listen 443 ssl; - This says respond to web requests on port 443 (default ssl port)
ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem; - this is the location of the PEM full certificate chain that Let's Encrypt generates for you.
ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem; - the location of the private key that Let's Encrypt generated for you. Necessary for SSL encryption to work
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - This limits what SSL protocols are served and accepted by the web server. As protocols get hacked this will need to get updated.
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH'; - this is the latest list that I found of secure encryption ciphers that WILL NOT work on older browser like IE 6.
ssl_prefer_server_ciphers on; - This tells clients that if you are not running my ciphers, you can't connect - LOSER
ssl_session_cache shared:SSL:10m; - honestly don't know what this does. Guessing performance caching for repeat clients.
ssl_dhparam /etc/ssl/certs/dhparam.pem; - This is the wizard that helps get the "A" SSL rating. By default if you don't include this I got a "B" rating from Qualys SSL Labs (https://ssllabs.com/). The reasoning is that Diffie Hellman algorithm is more secure when establishing connections. The unfortunate downside to this is that you need to generate the DH pem file. On my Pi 2 this took an hour, but I read in some places it took over 24 hours. It is random so it will take as long as it takes.
To generate that pem file:
cd /etc/ssl/certs/
sudo openssl dhparam -out dhparam.pem 4096
sudo openssl dhparam -out dhparam.pem 4096
You can choose a 2048 bit key which will go MUCH faster, but that is up to you. Now don't forget to set up a cron job (as root) to try to renew your SSL certs DAILY. They recommend that you try to renew it twice a day in case their servers are not available. There is no penalty for trying to renew a certificate and it doesn't count towards rate limiting.
sudo crontab -e
In the file add an entry like this:
0 1,13 * * * /home/pi/certbot/certbot-auto renew >> /var/log/letsencrypt-renew.log
Note: after a successful update you will need to restart Nginx. You can just restart it during your maintenance window or just do a quick restart a few minutes after you try to renew your cert. I elected for a restart at 2am every day. Just add it to your root crontab and you are all set:
0 2 * * * service nginx restart
Restart Nginx again and test to make sure you can hit your page via SSL. Finger crossed. If you can, now check your rating on https://ssllabs.com. If you are bold let them post your results on their leader board. I'm not that bold. Right now I have an "A" rating and if any new fun SSL bugs come out, I'll have to update my cipher list. Welcome to DevOps!
Subscribe to:
Posts (Atom)