One-liner: Decompress file and diff against another file
Dec 8th, 2016 by miki

Below is a handy shell one-liner for comparing the decompressed contents of a compressed file against a plain file. The purpose here is to test whether the compressed file is actually derived from compressing the plain file. This particular example is from a real life situation where log rotation by cron.daily on a Ubuntu server had begun failing. The situation is thought to be the result of an interrupted logrotate execution leaving a compressed (but non-rotated) intermediate file in the file system that prevented further log rotation on subsequent executions.

The command constructs a pipe between two process, specified using the | symbol (vertical line), to send the decompressed contents from gunzip stdout to stdin of the compare tool. The example uses diff as compare tool which is suitable for textual contents. You could use f.x. cmp instead if the contents is binary. Both diff and cmp interprets the file name “-” as meaning that input should be read from stdin.

$ gunzip --stdout /var/log/syslog.1.gz | diff --report-identical-files - /var/log/syslog.1
Files - and /var/log/syslog.1 are identical

The above uses long options for clarity, in a real life situation you would probably be using short options instead. That means -c instead of –stdout and s instead of –report-identical-files.

$ gunzip -c /var/log/syslog.1.gz | diff -s - /var/log/syslog.1
Files - and /var/log/syslog.1 are identical


Raw HTTP session with telnet
Jul 21st, 2010 by miki

Once in a while it is useful to dismiss abstractions and layers that makes daily routines easier and take the raw approach. Like when debugging a software problem that doesn’t make sense, it is nice to see the underlying basic stuff is behaving nicely, to better be able to locate where the unexpected occurs.

In the IP world of the internet, the swiss army knife for debugging interprocess communications in a totally protocol agnostic way is called ‘telnet’. Telnet opens up a communication channel between your local computer and a daemon/server on a specific port on a specific IP address. Then it gets out of the way for you to talk directly to the daemon in clear text.

Knowledge of how to interact using a specific protocol can be very useful to check server availability and functionality. All common protocols in use on the internet (like DNS, HTTP, SMTP, POP3, IMAP, XMPP etc.) can be debugged like this, because all of them transfers data in clear text (or at least initiates other transfer types from a clear text session).

Below is a basic HTTP session to my web server using telnet command line on a Linux box. Full specifications for the HTTP protocol can be found in IETF RFC2616. I keep forgetting this, and end up digging around for it when needed, therefore this blog post.

Red text is local text input by me. Blue text is local text by telnet application. Green text is server response.

miki@khandro:~$ telnet 80
Connected to
Escape character is ‘^]’.
GET /index.php/2010/06 HTTP/1.1

HTTP/1.1 200 OK
Date: Wed, 21 Jul 2010 09:19:13 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” ““>

<… remainder of HTML document dropped …>

We are asking the server at port 80 (default http port) for the document at path /index.php/2010/06 (“GET /index.php/2010/06”). Notice the two CRLF characters after the Host header field, this indicates to the server that the request header is done, and that it should begin parsing header and send its response. Also notice that even though you tell the telnet program on the commandline that you want to access, you have to tell it again to the server in th HTTP Host field. Thats because telnet is only concerned about the IP address of the server, it resolves to the IP through DNS and forgets about it. From the servers point of view, it needs to know which of its virtual hosts you want to talk to, cause one server application on one port on one ip can potentially host thousands of separate websites (virtual hosts).

»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2016 Mikini Services