Using rsync over SSH to backup or transfer files

In this tutorial we will cover the steps needed to copy files with rsync over SSH in Debian or Ubuntu platform. While having amazing Catuai coffee from Honduras I learned the following.

Rsync is a great tool that allows you to transfer and synchronize data between servers. The command can be used over SSH which encrypts the connection. It also provides large amount of options which can be used such as archive mode, backup mode, data compression during the transfer etc.

If rsync is not included in your distro you can easily install it using (Debian/Ubuntu way):

sudo apt install rsync

In order to make sure that you will be able to transfer files from/to the remote server using rsync over SSH you can first try to establish an SSH connection:

ssh user1@<yourServerIP or name>

The recommended way to connect to your server is by using keys. To generate keys run in the terminal:

ssh-keygen -f ~/.ssh/id_rsa

and then:

cat ~/.ssh/id_rsa.pub

Copy this key to your clipboard and login to your destination server. Place this SSH key into your ~/.ssh/authorized_keys file. If your SSH folder does not exist, create it manually:

mkdir ~/.ssh 
chmod 0700 ~/.ssh 
touch ~/.ssh/authorized_keys 
chmod 0644 ~/.ssh/authorized_keys

Rsync files over

rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /root/bigfile.txt <yourServerIP or name>:/root/

If you are using a different user, for example “username” then you would have to append it in front of destination server. Make sure to have your public key in that user’s ~/.ssh/authorized_keys file:

rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /root/bigfile.txt username@<yourServerIP or name>:/

The newer versions of rsync are configured to use SSH as default remote shell so you can omit the -e ssh option.

For simpler rsync file transfers you can do any of the following:

For example, to transfer a single file /opt/myfile.zip from the local system to the /var/www/ directory on the remote Linux system with IP 163.90.162.123 you would run:

rsync -a /opt/myfile.zip <yourUser>@163.90.162.123:/var/www/

Replace <yourUser> with your actual username for your remote Linux server. The -a option stands for archive mode which will syncs directories recursively, transfer special and block devices, preserve symbolic links, modification times, group, ownership, and permissions. Very cool!

To transfer files from the remote Linux server to your local machine just do the opposite:

rsync -a <yourUser>@163.90.162.123:/var/www/ /opt/myfile.zip

Contact me if you have any questions. Thanks!

Common HTTP response status codes to Understand

Hola! While having a great cup of Caturra coffee from Panama I decided to put together the most common HTTP response status codes in a simple list.

HTTP response status codes indicate whether a specific HTTP request has been successfully completed. Responses are grouped in five classes:

  1. Informational responses (100–199),
  2. Successful responses (200–299),
  3. Redirects (300–399),
  4. Client errors (400–499),
  5. and Server errors (500–599).

I’m not going to list all examples, but I will list the most common HTTP response status below:

Successful HTTP responses

200 OK

The request has succeeded. The meaning of the success depends on the HTTP method:

  • GET: The resource has been fetched and is transmitted in the message body.
  • HEAD: The entity headers are in the message body.
  • PUT or POST: The resource describing the result of the action is transmitted in the message body.
  • TRACE: The message body contains the request message as received by the server

Redirection message

301 Moved Permanently

The URL of the requested resource has been changed permanently. The new URL is given in the response.

307 Temporary Redirect

The server sends this response to direct the client to get the requested resource at another URI with same method that was used in the prior request. This has the same semantics as the 302 Found HTTP response code, with the exception that the user agent must not change the HTTP method used: If a POST was used in the first request, a POST must be used in the second request.

308 Permanent Redirect

This means that the resource is now permanently located at another URI, specified by the Location: HTTP Response header. This has the same semantics as the 301 Moved Permanently HTTP response code, with the exception that the user agent must not change the HTTP method used: If a POST was used in the first request, a POST must be used in the second request.

Client Error HTTP Responses

400 Bad Request

The server could not understand the request due to invalid syntax.

401 Unauthorized

Although the HTTP standard specifies “unauthorized”, semantically this response means “unauthenticated”. That is, the client must authenticate itself to get the requested response.

403 Forbidden

The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Unlike 401, the client’s identity is known to the server.

404 Not Found

The server can not find the requested resource. In the browser, this means the URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource itself does not exist. Servers may also send this response instead of 403 to hide the existence of a resource from an unauthorized client. This response code is probably the most famous one due to its frequent occurrence on the web.

408 Request Timeout

This response is sent on an idle connection by some servers, even without any previous request by the client. It means that the server would like to shut down this unused connection. This response is used much more since some browsers, like Chrome, Firefox 27+, or IE9, use HTTP pre-connection mechanisms to speed up surfing. Also note that some servers merely shut down the connection without sending this message.

Server error HTTP responses

500 Internal Server Error

The server has encountered a situation it doesn’t know how to handle.

502 Bad Gateway

This error response means that the server, while working as a gateway to get a response needed to handle the request, got an invalid response.

503 Service Unavailable

The server is not ready to handle the request. Common causes are a server that is down for maintenance or that is overloaded. Note that together with this response, a user-friendly page explaining the problem should be sent. This responses should be used for temporary conditions and the Retry-After: HTTP header should, if possible, contain the estimated time before the recovery of the service. The webmaster must also take care about the caching-related headers that are sent along with this response, as these temporary condition responses should usually not be cached.

505 HTTP Version not supported

The HTTP version used in the request is not supported by the server.

Could not set the file size of ./ibtmp1. Probably out of disk space

While having my daily specialty coffee from Latin America I got this error for one of my docker containers running MySQL. ERROR: ‘Could not set the file size of ./ibtmp1’. Suddenly the container failed to start and I could not understand why!

I reviewed the logs for my container:

docker logs <my_container_name>
2020-07-31T15:45:10.209380Z 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
2020-07-31T15:45:10.209437Z 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
2020-07-31T15:45:10.209486Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-07-31T15:45:10.725388Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-07-31T15:45:10.725530Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-07-31T15:45:10.725673Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-07-31T15:45:10.725770Z 0 [ERROR] Failed to initialize builtin plugins.
2020-07-31T15:45:10.725816Z 0 [ERROR] Aborting
2020-07-31T15:45:10.725871Z 0 [Note] Binlog end
2020-07-31T15:45:10.725996Z 0 [Note] Shutting down plugin 'CSV'
2020-07-31T15:45:10.727174Z 0 [Note] mysqld: Shutdown complete

Something strange ‘Could not set the file size of ‘./ibtmp1′. Probably out of disk space’

My computer disk space was fine with more than 20GB free space. But, then after a walk around my hood in NYC Moments I discovered this error was referring to my virtual space hard disk created by the docker engine which is used in the background. This was not my local computer’s hard disk space.

This solution worked for me:

docker-compose down
docker system prune
docker-compose up

I ran those three commands. This will prune all containers that not in used and might also reset your current containers. In my case I had to re-create MySQL database and tables. I need to make this better. Hope this solution for ‘Could not set the file size of ./ibtmp1’ error works for you. Remember to check out some cool coffee mugs and T-Shirt designs on my sister website

How to mount EXT Linux partition on MacOS

The other day while having a great cup of Latin America coffee from El Salvador I found a way to mount my EXT Linux partition I have on an external backup WD drive. I didn’t know that MacOS could not mount it natively even when both Linux and MacOS come from the same mother UNIX.

Homebrew came to the rescue! First, make sure you have it installed. Then run these couple of commands in order to install the package ext4fuse. Open your terminal and run these two commands:

brew cask install osxfuse
brew install ext4fuse

Plug in the hard disk drive or storage device whose partition you want to mount and in the terminal run:

diskutil list
diskutil list command results

This will give you a list of storage devices attached to your computer. Lets say your device is disk2S1. Then you run:

sudo ext4fuse /dev/disk2s1 ~/tmp/MY_EXT_PARTITION -o allow_other

MY_EXT_PARTITION is only a sample name. You can name it anything you want. You need sudo privileges to run this command.

Now, you can use Finder to navigate to your /tmp directory and find the mounted partition. Or in the terminal do:

open ~/tmp/

Remember, this methods will give you READ ONLY rights to the mounted ext partitions. You cannot make changes to it. Can you view the files and copy them to your MacOS and then make any changes. Homebrew is very handy and help me mount my EXT Linux partition I have laying around many external hard drives.

Hope this little tutorial was helpful. Contact me if you have any questions or need help.

grep Command in Linux Standard Examples

grep command is your friend for searching mainly text. It searches a given file for lines containing a given string or words. The grep command should be one of the most used commands and you should get familiar with it. I’m doing the same.

Below are some standard grep commands:

Search for any instance of the word ‘coffee’ in a file do:

grep 'coffee' filename

To perform a case sensitive search use the -i flag:

grep -i 'coffee' filename

The above command will give you any words matching ‘Coffee’, ‘coffee’, ‘cOffEe’. You get the idea. Latin America specialty coffee is fantastic!

To find all occurences of the word ‘new york city’ in a given directory and all its subdirectories:

grep -R 'new york city' .

Using the ‘.’ (dot or period) tells Linux to search in the current location.

I hope this little tutorial gives you some inspiration to learn more about the grep command. Check the man pages in your Linux distro:

man grep

Or install cheat in Ubuntu using apt or snap

apt install cheat or snap install cheat

Cheat works great for me. It gives me a brief summary of most Linux commands and its easy to use. Just run it your terminal for example:

cheat grep

How to display time in Python

I have been using Flask to teach myself some coding and develop a web application for people with high blood pressure to track their daily blood pressure readings.

I wanted to find a way to automatically display the time when a user takes a blood pressure reading. It was quite simple.

Use strftime() to display Time and Date

The strftime() method returns a string displaying date and time using date, time or datetime object.

In your .py file you have to import time. Below are some examples to print time in different formats:

print (time.strftime("%H:%M:%S")) - Displays time in 24hr format.
print (time.strftime("%I:%M:%S")) - Displays time in 12hr format.

See more examples in the Python official documentation. Enjoy!

Grab a coffee mug from my sister website NYCmoments.nyc

Setting Up Static IP on Ubuntu Server 20.04 LTS

Ubuntu Server 20.04 LTS uses the tricky Netplan for network configuration by default. Network configuration is a bit more tricky, but still good. The default Netplan network configuration file on Ubuntu 20.04 LTS server is /etc/netplan/00-installer-config.yaml.

First, find the network interface name which you want to configure a static IP address with the following command:

$ ip a
ip a command output
ip a command output

As you can see my network interface name is eth0@if8, it will be different for you.

Now, go brew a good cup of coffee you have to make sure that the network interface is not managed by CloudInit.

For that, open the configuration file with your favorite editor /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg with the following command:

$ sudo vim /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg

Make sure the configuration file contains the following line:

network: {config: disabled}

Now, open the Netplan configuration file using your favorite editor (I like using vim) /etc/netplan/00-installer-config.yaml with the following command:

sudo vim /etc/netplan/00-installer-config.yaml

To assign a static IP address to the network interface (in my case, your network interface name will be different) eth0@if8, remove everything from the Netplan configuration file /etc/netplan/00-installer-config.yaml and type in the following lines.

network:
  version: 2
  ethernets:
    eth0@if8:
      addresses: [192.168.1.160/24]
      gateway4: 192.168.1.2
      nameservers:
        addresses: [192.168.1.2, 8.8.8.8]

Now, to make sure that the configuration file does not have any syntax errors, run the following command:

sudo netplan try

If everything is good you should see a message with something like “Warning: stopping systemd-networkd.service, but it can still be activated by:…..Press ENTER before the timeout to accept the new configuration”

The new network configuration should be accepted. To make changes permanent run the following command:

sudo netplan apply

Double check the IP address run again:

ip a

Enjoy! Contact me if you have questions or leave a comment on the comments section below.

If you like to learn more about Ubuntu Server you can get this book Ubuntu Unleashed 2019 Edition: Covering 18.04, 18.10, 19.04 (13th Edition).

How to optimize images for the web using GIMP

Optimizing images for the Web means reducing its file size which in turn will make it load faster on websites. Grab some specialty coffee and follow these simple instructions.

Reducing the image size with GIMP

To reduce the image size:

  1. Open image in GIMP.
  2. Click on Image in the toolbar, then select Scale Image.
  3. In the image size, change the width to 1920px (or your own width)
  4. Select Scale
  5. Select File in the toolbar. Then select Export As.
  6. Name to your file ending with a “.jpg”
  7. Select Export
  8. In dialog box that opens, change Quality to 60.
  9. Click Advanced Options.
  10. Change Subsampling to 4:2:0 (chroma quartered)
  11. Click Export

Feel free to adjust settings to meet your image quality needs.

Contact me if you have any questions.

Linux command line history tips

Many times I found myself typing history into the terminal and then copy/paste the command I was looking for. But, then I found the below tricks to help repeat that command faster and more efficiently.

Use Control+R

I found this to be the most beneficial for me. It will do a reverse search really fast! Just press enter when you see the command you need to retype.

Repeat the previous command quickly

This is useful if you know you recently typed a command. You can use the following key combinations:

  • Use the up arrow to view the previous command and press enter to execute it.
  • Type !! and press enter from the command line
  • Type !-1 and press enter from the command line.
  • Press Control+P will display the previous command, press enter to execute it

Execute a specific command from history

For example, you can display history as usual:

history | more
1  service apache2 restart
2  ifconfig
3  ls
4  cat /etc/hosts

If you want to repeat command #2, you will type !2 as show below:

# !2
eth0      Link encap:Ethernet  HWaddr 12:38:33:b7:99:96  
          inet addr:10.0.0.17  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::1038:33ff:feb7:9996/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:586486972 errors:0 dropped:0 overruns:0 frame:0
          TX packets:539727562 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:363967988525 (363.9 GB)  TX bytes:491630839370 (491.6 GB)

MySQL how to insert a row into a table

Inserting data into MySQL tables is easy, but I tend to forget little details once in a while. I was having the best espresso coffee and trying to remember this one. Below is how you insert a new row into a MySQL table.

Assume you created the following table:

CREATE TABLE IF NOT EXISTS mytable (
    mytable_id INT AUTO_INCREMENT,
    title VARCHAR(255) NOT NULL,
    start_date DATE,
    due_date DATE,
    priority TINYINT NOT NULL DEFAULT 3,
    description TEXT,
    PRIMARY KEY (task_id)
);

And then you want to insert a new row into this MySQL table. Run the command below:

INSERT INTO tasks(title,priority)
VALUES('Learn MySQL INSERT Statement',1);

You should get the below from MySQL:

1 row(s) affected

That’s all! Contact me if you need help. Write in the comments section if you have questions.