SIDEBAR
»
S
I
D
E
B
A
R
«
Automating wav to ogg conversion
Jun 6th, 2019 by miki

Needed some space on the disk.

Had some music, some of it were wav files.

Threw together this script to transmogify it into ogg using ffmpeg.

Statistics say that 24 wav files occupying 1,7 GiB was made into 24 ogg files occupying 105 MiB, a reduction to ~6% (du -h calculates 2^10 numbers but uses SI 10^3 prefixes..).

miki@filly:~/Music$ find . -iname "*.wav" -print0 | sed s/.wav/.ogg/g |du --files0-from=- -h -c|tail -1
105M	total
miki@filly:~/Music$ find . -iname "*.wav" -print0 | sed s/.wav/.wav/g |du --files0-from=- -h -c|tail -1
1,7G total
miki@filly:~/Music$ find . -iname "*.wav" -print0 | sed s/.wav/.ogg/g |du --files0-from=- -h |wc -l
24
miki@filly:~/Music$

Script

Install prerequisite ffmpeg and ffprobe utilities on apt distributions by doing “apt install ffmpeg”.

BEWARE: script deletes any wav files in current directory and below without warning if conversion to ogg is fine according to ffmpeg return code and stdout/err.

Manual confirmation of all deletions can be enabled by uncommenting line 8 (CONFIRM=..).

It could (very) easily be amended for conversion between any container or compression format supported by ffmpeg by replacing “wav” and “ogg” in glob pattern and OGG variable definition (for completeness also in echo output…).

miki@filly:~/wavtest$ cat ~/hometools/wav2ogg.sh
#! /bin/bash
#
# Make .ogg out of all .wav in cwd and below, remove .wav!
#
# When converting remove only if conversion is ok.
# ffprobe any present .ogg and check if playtime is same as wav, remove only if all is good.

#CONFIRM=-i # confirm deletion?

REQS="ffmpeg ffprobe"
which $REQS  >/dev/null
if [ ! $? -eq 0 ]; then
    echo We need $REQS to do stuff!
    exit 1
fi

shopt -s globstar nullglob nocaseglob # enable 2-star magic
for f in **/*.wav; do
    OGG=$(dirname "$f")/$(basename "$f" wav)ogg
    echo -n "$f: "
    if [ ! -f "$OGG" ]; then
        echo -en "\e[33mno ogg\e[0m"
        CONVERT=1
    else
        echo -en "\e[32malready there\e[0m"
        if [ ! "$(ffprobe "$f" 2>&1|grep Duration|cut -d, -f1)" = "$(ffprobe "$OGG" 2>&1|grep Duration|cut -d, -f1)" ]; then
            echo -en " - \e[31mogg corrupt\e[0m"
            CONVERT=1
            rm $CONFIRM "$OGG"
        else
            OGGOK=1
        fi
    fi
    if [ $CONVERT ]; then
        echo -en " - \e[33mconverting\e[0m"
        OUT=$(ffmpeg -loglevel error -i "$f" "$OGG" 2>&1)
        if [ ! $? -eq 0 ]; then
            echo -e " \e[31mfatal error!\e[0m\nffmpeg output was:\n\n$OUT\n"
            rm $CONFIRM "$OGG"
        elif [ ! -z "$OUT" ]; then
            echo -e " \e[31mnon-fatal error!\e[0m\nffmpeg output was:\n\n$OUT\n"
        else
            OGGOK=1
        fi
    fi
    if [ $OGGOK ]; then
        echo -e " - all good removing wav"
        rm $CONFIRM "$f"
    fi
done
miki@filly:~/wavtest$ 

Usage and Output

Before

miki@filly:~/wavtest$ ls -lR
.:
total 59964
drwxrwxr-x 3 miki miki     4096 Jun  6 22:33 one
-rw-rw-r-- 1 miki miki 61390864 Jun  6 22:32 Sordid Affair (Man Without Country Version).wav

./one:
total 30720
drwxrwxr-x 3 miki miki     4096 Jun  6 22:34 two
-rw-rw-r-- 1 miki miki 31451304 Jun  6 22:30 Vision Man MASTER.wav

./one/two:
total 55352
-rw-rw-r-- 1 miki miki 56675660 Jun  6 22:34 Bye Barat - Going For Broke (UNKWON´s Autumn Remix) LIM_02.wav
drwxrwxr-x 2 miki miki     4096 Jun  6 22:34 three

./one/two/three:
total 375112
-rw-rw-r-- 1 miki miki 384108056 Jun  6 22:30 Rigoremix - Pornographe - Final2.wav
miki@filly:~/wavtest$ du -hs .
509M	.
miki@filly:~/wavtest$

During

In real life, also in technicolor!

miki@filly:~/wavtest$ ~/hometools/wav2ogg.sh 
one/two/Bye Barat - Going For Broke (UNKWON´s Autumn Remix) LIM_02.wav: no ogg - converting - all good removing wav
one/two/three/Rigoremix - Pornographe - Final2.wav: no ogg - converting - all good removing wav
one/Vision Man MASTER.wav: no ogg - converting - all good removing wav
Sordid Affair (Man Without Country Version).wav: no ogg - converting - all good removing wav
miki@filly:~/wavtest$

After

miki@filly:~/wavtest$ ls -lR
.:
total 4952
drwxrwxr-x 3 miki miki    4096 Jun  6 22:35 one
-rw-rw-r-- 1 miki miki 5066460 Jun  6 22:35 Sordid Affair (Man Without Country Version).ogg

./one:
total 2528
drwxrwxr-x 3 miki miki    4096 Jun  6 22:34 two
-rw-rw-r-- 1 miki miki 2580752 Jun  6 22:35 Vision Man MASTER.ogg

./one/two:
total 4680
-rw-rw-r-- 1 miki miki 4785594 Jun  6 22:34 Bye Barat - Going For Broke (UNKWON´s Autumn Remix) LIM_02.ogg
drwxrwxr-x 2 miki miki    4096 Jun  6 22:35 three

./one/two/three:
total 19456
-rw-rw-r-- 1 miki miki 19922936 Jun  6 22:35 Rigoremix - Pornographe - Final2.ogg
miki@filly:~/wavtest$ du -hs .
31M .
miki@filly:~/wavtest$

Metadata

ffmpeg does some nice metadata conversions out of the box;

miki@filly:~/Music$ ffprobe ~/"wavbak/Delayscape  A Short While Rigolin Edit Audaccity.wav" 2>&1| grep Input -A 1000
Input #0, wav, from '/home/miki/wavbak/Delayscape  A Short While Rigolin Edit Audaccity.wav':
  Metadata:
    title           : Delayscape - A Short While (Rigolin Edit)
    artist          : rigolin
    comment         : Love Melody
    date            : 2018
  Duration: 00:06:33.97, bitrate: 3072 kb/s
    Stream #0:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, 2 channels, flt, 3072 kb/s
miki@filly:~/Music$ ffprobe "./Delayscape - Heinz Beauvaix/Soundcloud/Delayscape  A Short While Rigolin Edit Audaccity.ogg" 2>&1| grep Input -A 1000
Input #0, ogg, from './Delayscape - Heinz Beauvaix/Soundcloud/Delayscape  A Short While Rigolin Edit Audaccity.ogg':
  Duration: 00:06:33.97, start: 0.000000, bitrate: 99 kb/s
    Stream #0:0: Audio: vorbis, 48000 Hz, stereo, fltp, 112 kb/s
    Metadata:
      ENCODER         : Lavc56.60.100 libvorbis
      TITLE           : Delayscape - A Short While (Rigolin Edit)
      ARTIST          : rigolin
      comment         : Love Melody
      DATE            : 2018
miki@filly:~/Music$

Source Material

Silent browser redirect on invalid certificate
Nov 15th, 2018 by miki

Hit an odd browser behaviour today.

Trawling through some of those nice and dandy terms for a service (no I won’t tell) I followed a link and suddenly hit an “insecure connection” message from Firefox.

Examining the certificate using the usual “openssl s_client” and “openssl x509” tools surely enough revealed that the served certificate didn’t include the second-level but only on the third-level www sub-domain. Strangely enough I discovered that when entering the same URL directly into the address bar of Firefox the connection was somehow redirected to the www sub-domain and loaded fine without any complaints from Firefox.

Looking into what was happening on the wire using wget directed to not check the certificate (–no-check-certificate) and displaying server responses (–server-response/-S) revealed that the server behind the misconfigured certificate was aiming to issue a HTTP 301 redirect to the valid www sub-domain of the site (actual site replaced by foo.bar and localhost ip);

$ wget –server-response –no-check-certificate –output-document=/dev/null https://foo.bar
–2018-11-15 18:59:14– https://foo.bar/
Resolving foo.bar (foo.bar)… 127.0.0.1
Connecting to foo.bar (foo.bar)|127.0.0.1|:443… connected.
WARNING: no certificate subject alternative name matches
requested host name ‘foo.bar’.
HTTP request sent, awaiting response…
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Location: https://www.foo.bar/
Date: Thu, 15 Nov 2018 17:59:13 GMT
Content-Length: 152
Location: https://www.foo.bar/ [following]

Apparently Firefox’s behaviour in this scenario differs depending on the method in which the user is supplying the URL, in some cases silently ignoring the TLS/SSL warning. A little more experimentation revealed that it was actually not the source of the URL, link followed versus one entered in address bar, that made any difference. Instead it is the presence of a trailing slash on the URL that triggers the silent suppression of the serious red flag that the second-level domain asked to be fetched is not present in the served certificate. I was fooled by the fact that when entering URLs in the address bar Firefox suggests ending the URL on a slash (‘/’) if it is not already present. Manually removing this when editing in the address bar also makes Firefox display the security warning. Alas, the link I originally followed was also missing the trailing slash and behaved as expected by throwing me into a security warning.

The whole “feature” of silently ignoring a security issue seemed very odd to me, but a bit of searching revealed that this was apparently championed by Google Chrome a couple of years ago. It is described in this servertastic post which also directs to a discussion on Twitter, with some key points replicated below, about its presence in Chrome and confirmation from a Chrome team member that the behaviour is intended.

The thread ends with a guy who had dug up the source code of the feature in Chromium (on which Chrome is based) known as “SSLCommonNameMismatchHandling” in file browser/ssl/common_name_mismatch_handler.cc (see all mentions across code base).

This has obviously also trickled down into Firefox, however, not much mention of that is to be found. Not even traces of it by some quick searches of the mozilla-central codebase. Some day I promise (really!) to dig through all source of Mozilla and hunt down the implementation, but for now I’ll revert to a bit of practical experimentation showing the behaviour in the different browsers and operating systems I happen to have access to at the moment;

  • Ubuntu 16.04
    • Firefox 62.0.3: warning, with trailing ‘/’: No warning
    • Chromium 69.0.3497.81: no warning, with trailing ‘/’: no warning
  • Windows Server 2016
    • Firefox 62.0.2: warning, with trailing ‘/’: no warning
    • Google Chrome 70.3538.102: no warning, with trailing ‘/’: no warning
    • Internet Explorer 11.2580.14393.0: warning, with trailing ‘/’: warning
  • Windows Server 2008
    • Firefox 59.0.3: warning, with trailing ‘/’: warning
    • Google Chrome 70.3538.102: no warning, with trailing ‘/’: no warning
    • Internet Explorer 11.0.9600.19080: warning, with trailing ‘/’: warning

So somewhere between 59.0.3 and 62.0.2 Firefox also implemented a policy of silently accepting invalid certificates when certain non-obvious criteria is met (is the redirect actually followed and certs checked, or is “www.” just prefixed?), but this happens only when the URL ends on a slash. Go figure…

Raw HTTP session with telnet
Jul 21st, 2010 by miki

Once in a while it is useful to dismiss abstractions and layers that makes daily routines easier and take the raw approach. Like when debugging a software problem that doesn’t make sense, it is nice to see the underlying basic stuff is behaving nicely, to better be able to locate where the unexpected occurs.

In the IP world of the internet, the swiss army knife for debugging interprocess communications in a totally protocol agnostic way is called ‘telnet’. Telnet opens up a communication channel between your local computer and a daemon/server on a specific port on a specific IP address. Then it gets out of the way for you to talk directly to the daemon in clear text.

Knowledge of how to interact using a specific protocol can be very useful to check server availability and functionality. All common protocols in use on the internet (like DNS, HTTP, SMTP, POP3, IMAP, XMPP etc.) can be debugged like this, because all of them transfers data in clear text (or at least initiates other transfer types from a clear text session). Full specifications for the HTTP protocol can be found in IETF RFC2616. I keep forgetting this, and end up digging around for it when needed, therefore this blog post.

Interactively using Telnet

Below is a basic HTTP session to my web server www.mikini.dk using telnet on the command line.

Red text is local text input by me. Blue text is local text by telnet application. Green text is server response.

$ telnet www.mikini.dk 80
Trying 92.61.152.47…
Connected to 92-61-152-47.static.servage.net.
Escape character is ‘^]’.
GET /index.php/2010/06 HTTP/1.1
Host: www.mikini.dk

HTTP/1.1 200 OK
Date: Wed, 21 Jul 2010 09:19:13 GMT
Server: Apache
X-Pingback:
http://www.mikini.dk/xmlrpc.php
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

2bd2
<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd“>

<… remainder of HTML document dropped …>

We are asking the server at www.mikini.dk port 80 (default http port) for the document at path /index.php/2010/06 (“GET /index.php/2010/06”). Notice the two CRLF characters after the Host header field, this indicates to the server that the request header is done, and that it should begin parsing header and send its response. Also notice that even though you tell the telnet program on the commandline that you want to access www.mikini.dk, you have to tell it again to the server in th HTTP Host field. Thats because telnet is only concerned about the IP address of the server, it resolves www.mikini.dk to the IP 92.61.152.47 through DNS and forgets about it. From the servers point of view, it needs to know which of its virtual hosts you want to talk to, cause one server application on one port on one ip can potentially host thousands of separate websites (virtual hosts).

One-liner using netcat (nc)

Below is the same session using nc (netcat) as a one-liner (wow, old PHP at Hostinger, better get that move to a self-administered box going).

$ echo -ne "GET /index.php/2010/06 HTTP/1.1\r\nHost:www.mikini.dk\r\n\r\n"|nc www.mikini.dk 80 | head -10
HTTP/1.1 200 OK
Date: Thu, 05 Jul 2018 16:30:07 GMT
Server: Apache
X-Powered-By: PHP/5.5.35
Link: <http://www.mikini.dk/wp-json/>; rel="https://api.w.org/"
Content-Length: 32356
Content-Type: text/html; charset=UTF-8

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

$

2018-07-05: nc command line added

»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2019 Mikkel Kirkgard Nielsen, contents CC BY-SA 4.0