SIDEBAR
»
S
I
D
E
B
A
R
«
When email in a .msg file is not ASCII
Sep 18th, 2017 by miki

Got myself an exported message from (apparently) Exchange/Outlook in a file with .msg extension. My initial thought was that this was just a plain ASCII email (seem to remember having handled .msg files as such at some point), but looking at it as text exposed a load of binary and the “file” utility reported it being of type “Composite Document File V2 Document, Cannot read section info”.

What is it?

Some searching reveals .msg is in a proprietary format called “Outlook Item (.msg) File Format” (or formally, MS-OXMSG, find the specification  in the MSDN document entitled “[MS-OXMSG]: Outlook Item (.msg) File Format“). This format is related to the CFBF ([MS-CFB]: Compound File Binary File Format).

Check out what Library of Congress has to say about .msg files.

Interpreting it

libgsf

The GNOME project has developed an open source library and some tools for interacting with files of this type, check out libgsf documentation at developer.gnome.org/gsf/ and source code at github.com/GNOME/libgsf. Or install the package named libgsf-bin in Debian/Ubuntu for access to the “gsf” binary (man page) that can inspect and dump contents from a .msg file. This is quite rudimentary if you just want to read the content.

MSGViewer (java)

MSGViewer is a stand alone java application that can display/convert different mail-format, including .msg. On GitHub I found a fork which claims to provide further features than the original.

msgconv (perl)

msgconv (github source) is a perl script that converts a .msg file into standard RFC822/2822/5322 format .eml (email) format. Install as package “libemail-outlook-message-perl” on Debian/Ubuntu.

$ file test.msg
test.msg: Composite Document File V2 Document, Cannot read section info
$ msgconvert test.msg
$ file test.eml
test.eml: UTF-8 Unicode text, with very long lines, with CRLF line terminators

I prefer the approach of using a standard format for storing data, so I ended up accepting the .eml from msgconv which I propagated to other recipients of the file.

ToS humor: Tagalog speakers at Meetup.com
Jul 4th, 2017 by miki

A little humor is possible even when filtered through the eyes of lawyers. I’ll try to make a write-up on the ones I stumble across when reading ToS’es in the future.

The first is from the event site meetup.com which has embedded a sentence in the Tagalog language in the “Translation” section of their most recent edition of the site’s Terms of Service agreement (I was prompted to agree today even though the last update was on March 28, more than 3 months ago).

The text is “Umaasa kami na maintindihan mo” which according to Google Translate translated to “We hope you understand it“.

Even though they provide the previous editions for reference (it ought be illegal not to) I haven’t yet agreed to it as there is not an easy way to compare, it is a rather long read and there are non-trivial changes. I’d rather not use any service than to just agree carelessly to ToS updates (no, that is not a personality flaw).

 

Teknologi-danskere: Finn Hendil – Prøvebilledet
Apr 6th, 2017 by miki

Hvis du har mere end 15 leveår på bagen, har du nok set dette billede før:

Farveprøvebillede fra Phillips PM5544

Måske ved du også at en dansker har designet dette tidligere så berømmede prøvebillede? Det var ukendt for mig, før jeg læste dette Facebook opslag fra Universe Science Park (tidligere Danfoss Universe).

Men den danske ingeniør Finn Hendil, som arbejdede for Phillips i Danmark, var ham der i 1966 konstruerede grundlaget for dette mønster til test af farvefjernsyn, som i de følgende mange år blev solgt i apparaterne PM5540 og PM5544. De anvendtes til generering af prøvebilleder til udsendelse direkte fra TV-kanaler og som testudstyr hos fjernsynsproducenter.

Historien som fortalt af Erling Madsen i publikationen “Fjernsynets historie fra 1926 til 2012 – og lidt efter …“.

I Jenagade på Amager havde Philips en udviklingsafdeling og en produktion, som siden 1930’erne havde skiftet mellem bl.a. radioer, dyremad (under 2. verdenskrig), barbermaskiner, kanal-
vælgere til fjernsyn og radionavigation til lystsejlere samt mobiltelefoner. Ingeniør Finn Hendil ledede udviklingsafdelingen, og i 1966 fik han opgaven med at lave et prøvebillede, såteknikerne kunne se om farverne på farvefjernsynet stod rigtigt. Det blev siden dét farveprøvebille de, som blev standarden verden over og stadig bruges. Det er endda også udkommet på puder og t-shirt – og det var med i dødsannoncen, da Finn Hendil døde i januar 2011; en måned før han ville være fyldt 72 år.

 

Monokromt prøvebillede fra forløberen Phillips PM5540

Stallman in 2012: Denmark supposedly a free country, still valid
Mar 3rd, 2017 by miki

Stumbled upon this slightly dated talk by Richard M. Stallman (aka. RMS) of GNU and FSF fame, in which my home country of Denmark is sadly referenced as only a “supposedly free country”.

Transcript

“But censorship is wrong, of course, whether it is done on the internet or not. We used to think that the internet would protect us from censorship because it was too hard to censor the internet. But thanks to the effort of various companies in the US, The UK, France and so on, it is now possible for governments to censor the internet and also surveil it completely, they just need to put enough effort in. And this is not limited to obvious tyrannies such as China and Iran. We see a lot of supposedly free countries imposing censorship on the internet.

For instance, Denmark several years ago imposed filtering on the internet blocking a secret list of sites. The list was leaked and posted on WikiLeaks. Hooray for WikiLeaks! Whereupon Denmark blocked access to that page too. So everyone else could know what internet users in Denmark were blocked from seeing except those people.”

Sadly since this time it has not gotten any better. Most of the points RMS makes (the whole talk is worth a listen) are still valid and a grave concern from my perspective. The Danish internet (really DNS) blocking system has been broadened and the slippage that was feared has become a reality. Even though this issue has gotten some attention in the IT and rights communities the general public just doesn’t care.

The actual block is technically done through DNS blacklists that Danish ISP are legally required to implement. The list of blocked sites is available from the telecom trade organization “Telekommunikationsindustrien i Danmark” (English: Telecommunication’s Industry Association in Denmark) at teleindu.dk/brancheholdninger/blokeringer-pa-nettet/ and currently has 111 sites (csv) on active block.

As it being DNS based if you are impacted, workarounds do exist. However, my guess is that they will soon be able to actively shut down services physically located in Denmark.

Full talk

(starting at point of above transcript)

 

Generating passwords for Mosquitto MQTT broker using PHP
Jan 13th, 2017 by miki

Here is a delayed writeup of a my involvement in a question posted to the liberally licensed MQTT broker (server) Mosquitto’s developer list list about how to generate authentication tokens programmatically. It kicked the curious cat in me which propelled a journey into the backyards of the C code for the mosquitto_password tool which normally is used for this purpose. This resulted in the proof of concept PHP implementation outlined in my answer on the list which is reproduced below.

MQTT (once Message Queue Telemetry Transport) is a lightweight publish/subscribe protocol intended for communication between low power, low bandwidth embedded devices. These days it is commonly hyped as a holy grail in the religion of IoT. The protocol was originally developed by IBM but is now a standard overseen by the OASIS standardization organization which also has the OpenDocument standard (ODF, think Open/Libre-Office) under its wings. According to Wikipedia MQTT is used behind the scenes of Facebook Messenger, OpenStack and Amazon’s IoT services.

For further practical use of the concept outlined you would need to produce a random 16 byte base64 encoded salt to feed into the hasher, that could be done using something like; $salt_base64=base64_encode(openssl_random_pseudo_bytes(12));

If your need a shrink wrapped solution to this you could try to ping me.

Hi Srinivas.

On 2016-07-26 12:49, Srinivas Pokala wrote:
> Username successfully created using linux command with: 
> "mosquitto_passwd /etc/mosquitto/passwd guest".
> I need to create same with php or javascript how?

Looking at the source of mosquitto_passwd
(https://github.com/eclipse/mosquitto/blob/master/src/mosquitto_passwd.c)
basically all it does to generate the resulting line you see in the
password file is:

1) draw a random 12 byte binary salt
2) hash the combination of password and salt using sha512
3) write username, base64 encoded salt, base64 encoded hash in one line

A PHP implementation would use something like this ($salt is fixed for
demonstration purposes, it ought to be random in production);

---
$username="Bitten";
$password="Insect";
$salt_base64="spicychilinstuff";
$salt=base64_decode($salt_base64);
$hash=hash("sha512",$password.$salt, true);
$hash_base64=base64_encode($hash);
echo($username.":$6$".$salt_base64."$".$hash_base64."\n");
---

Comparing against mosquitto_passwd using a one-liner (uses the base64
salt from output to be able to correlate the two);

---
$ mosquitto_passwd -b ~/mosq_passwd_test Bitten Insect
$ cat ~/mosq_passwd_test
Bitten:$6$mfJ0Eq3rIDLKG33r$gkiIlz80JA6Pq9OtGhasIsx7L2vf0APdZH77+thmNW2Zp5vE1d/dAi5TjbfO9mZpKHLh38Oem1ic072rSO328g==

$ php -r '$username="Bitten"; $password="Insect";
$salt_base64="mfJ0Eq3rIDLKG33r"; $salt=base64_decode($salt_base64);
$hash=hash("sha512",$password.$salt, true);
$hash_base64=base64_encode($hash);
echo($username.":$6$".$salt_base64."$".$hash_base64."\n");'
Bitten:$6$mfJ0Eq3rIDLKG33r$gkiIlz80JA6Pq9OtGhasIsx7L2vf0APdZH77+thmNW2Zp5vE1d/dAi5TjbfO9mZpKHLh38Oem1ic072rSO328g==
---

As can be seen, the PHP generated password line are identical to the
mosquitto_passwd generated.

I have also successfully tested authentication against the mosquitto
broker with PHP generated users. One caveat is that the above can
generate a salt of arbitrary length, but the broker must see a 12 byte
binary salt (16 byte base64) or authentication will fail.

Note however, that this hasn't been tested on more than a few
username/password pairs, there might be other issues lurking.

Regards,
-- 
Mikkel
[Danish] Open source og LoRa er lidt vel store ord
Jan 6th, 2017 by miki

Min kommentar til LoRa og LoRaWANs åbenhed i forbindelse med artikel i Version2 med overskriften “TDC holder fast i proprietær IoT-standard – andre kører billig open source”, som nok mest minder om et bidrag til “open washing” af LoRa/LoRaWAN-teknologien. Se mere om open washing i denne blogpost fra Open Knowledge Foundation Denmark (Engelsk udgave).

Det er noget misvisende at snakke om at LoRa eller LoRaWAN er “open source”. Man er i hvert fald nødt til at skelne hårdt mellem LoRa og LoRaWAN.

LoRa specificerer radiomodulationen i luften, og det er en proprietær teknologi udviklet af virksomheden Semtech, som både har varemærkebeskyttet og patenteret den (EPO-patent);

LoRa is a proprietary spread spectrum modulation scheme that is derivative of Chirp Spread Spectrum modulation  (CSS)

Kilde: Semtech AN1200.22, LoRa(TM) Modulation Basics (http://www.semtech.com/images/datasheet/an1200.22.pdf, afsnit 1)

LoRaWAN er en åben protokol der anvender LoRa som transmissionsmedie, men som yderligere definerer hvordan flere enheder der alle kan snakke LoRa kan fungere sammen i et netværk. Den specificeres af et samarbejde mellem mange interessenter i LoRa Alliance, herunder også Semtech.

Q: What is LoRaWAN(TM)?
A: The LoRa modulation is the pshysical layer (PHY), and LoRaWAN is a MAC protocol for a high capacity, long range star network that the LoRa Alliance is standardizing for Low Power WideArea Networks (LPWAN).

Kilde: Semtech LoRa® FAQ (http://www.semtech.com/wireless-rf/lora/LoRa-FAQs.pdf, spgm. 3)

En mere teknisk beskrivelse kan findes her; https://www.lora-alliance.org/What-Is-LoRa/Technology. LoRaWAN-specifikationen selv er næsten frit tilgængelig, det er kun folk med offentlige email-adresser der ikke må få den.

For mig ser det dog ud som om Semtech og LoRa Alliance helt bevidst mudrer sondringen mellem LoRa og LoRaWAN, og synes det lugter af at det udelukkende er Semtech der kan og må producere silicium der implementerer LoRa. Selvom jeg ikke har kunnet finde steder det bliver sagt helt eksplicit.

Microchip og andre har produkter der snakker LoRa, men Semtech ser ud til altid at være med inde over, så jeg vil tro det stadig er en chip fra deres fabrik der ligger til grund for LoRa-funktionaliteten.

Der er dog folk der har kigget i sprækkerne på LoRa;

Spændende hvad der sker på dette felt. En mere fri og uafhængig infrastruktur i åbne licensbånd skal da være så velkommen.

Mikkel

[Danish] S&S: gemme data i Arduino ROM/Flash (PROGMEM / F())
Dec 21st, 2016 by miki

Mit svar på et spørgsmål i Facebook-gruppen Danske Arduino Entusiaster omkring Arduino ROM/Flash, PROGMEM og system-inklude-filer.

Spørgsmål

Hej er der en der ved hvor jeg kan hente dett lib. <avr/pgmspace.h> jeg skal bruge denne funktion PROGMEM
så jeg kan gemme et billede i Arduino uden SD kort
det kan være der er en der kender en anden måde at gøre det på.

Svar

pgmspace.h er en inklude-fil som er en del af c-biblioteket til AVR-arkitekturen (avr-libc). C-bibliotekets inklude-filer vil normalt ligge i kompilerens “system include”-sti (se GCC options -I og -isystem). Dermed kan den inkluderes blot med “#include <avr/pgmspace.h>”. Se evt. også Arduino-referencen på https://www.arduino.cc/en/Reference/PROGMEM.
 
Bemærk at PROGMEM ikke er en funktion, men en storage modifier (lager-modifikator) som fortæller kompileren at den kan placere en en given variabel i ikke-skrivbar lager (ROM/Flash). Der skal efterfølgende anvendes specielle funktioner til at læse data fra en sådan variabel (se referencen).
Arduino-frameworket har dog lavet en nem måde at placere konstant-strenge i Flash på (normalt lagres de i SRAM!), nemlig funktionen F() som kan anvendes direkte i f.eks. printf/write/print (Serial.print(F(“Waiting for connection”));)
 
Hvis du vil inspicere indholdet af pgmspace.h, kan du finde filen i Arduino IDE’ets installations-mappe under hardware/tools/avr/avr/include/avr/pgmspace.h. Det er ikke en man kan/skal redigere manuelt i, da den er tæt koblet med den binære kode i selve biblioteket.
 
Der findes også EEPROM-lager du sikkert vil kunne bruge til samme formål; https://www.arduino.cc/en/Reference/EEPROM

Se svaret på Facebook.

Den videre færd med F()

Da jeg ikke kunne finde en uddybende forklaring på F()-funktionen (som egentlig er en makro) i Arduino-dokumentationen (brugen nævnes meget kort i PROGMEM , Memory og Print), gravede jeg efterfølgende lidt rundt for at lære mere. I de sparsomme Arduino-eksempler er den anvendt udelukkende med konstante strenge, hvilket også viser sig at være et krav (eller i hvert fald noget der kan castes til const char *).

Makroen er defineret af Arduino-frameworket i filen hardware/arduino/avr/cores/arduino/WString.h (referencerne er ifht. min lokale installation af Arduino 1.6.9, pt. er nyeste 1.6.13) således:

#define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))

Altså parametren til F() bruges som parameter til PSTR() (progmem string, er mit bud på navn) som er en makro defineret i pgmspace.h fra avr-libc.

Dens funktion er at caste parametrens type til konstant streng-pointer med PROGMEM modifier;

#define PSTR(s) ((const PROGMEM char *)(s))

Skal vi se på hvad PROGMEM rent faktisk er, så finder vi endnu et sæt makroer der ender med at blive udviddet til kompiler-attributten  __progmem__, igen definieret i pgmspace.h (hardware/tools/avr/avr/include/avr/pgmspace.h):

#define PROGMEM __ATTR_PROGMEM__

#define __ATTR_PROGMEM__ __attribute__((__progmem__))

__progmem__ attributten er en instruks til kompileren (GCC) og linkeren om ved programmering/flashing af programmet at placere disse data i en sektion af hukommelsen der hedder “.progmem“. Se evt. mere om dette i GCC-kompilerens dokumentation. For hver AVR-chip kompileren understøtter er der eksakte definitioner af hvilke hukommelsesadresser .progmem ligger på for netop denne chip.

Dvs. når man i sin kode skriver F(“test”) får man i virkeligheden:

(reinterpret_cast<const __FlashStringHelper *>(((const __attribute__((__progmem__)) char *)(“test”)))

Altså en konstant streng der lagres i AVR-processorens progmem-sektion, og som returværdi får en pointer til en konstant instans af en klasse kaldet “__FlashStringHelper“. Denne klasse må være lavet sådan at den anvender de korrekte mekanismer til at læse fra progmem-området (måske mere om dette i en senere artikel). Arduinos funktion-bibliotek (Serial.print() mm.) er lavet således at de direkte kan tage en parameter af denne type som erstatning for en konstant-streng (og det er netop her Arduino-frameworket viser sin værdi ved at abstrahere sådanne kompleksiteter væk fra programmøren).

One-liner: Decompress file and diff against another file
Dec 8th, 2016 by miki

Below is a handy shell one-liner for comparing the decompressed contents of a compressed file against a plain file. The purpose here is to test whether the compressed file is actually derived from compressing the plain file. This particular example is from a real life situation where log rotation by cron.daily on a Ubuntu server had begun failing. The situation is thought to be the result of an interrupted logrotate execution leaving a compressed (but non-rotated) intermediate file in the file system that prevented further log rotation on subsequent executions.

The command constructs a pipe between two process, specified using the | symbol (vertical line), to send the decompressed contents from gunzip stdout to stdin of the compare tool. The example uses diff as compare tool which is suitable for textual contents. You could use f.x. cmp instead if the contents is binary. Both diff and cmp interprets the file name “-” as meaning that input should be read from stdin.

$ gunzip --stdout /var/log/syslog.1.gz | diff --report-identical-files - /var/log/syslog.1
Files - and /var/log/syslog.1 are identical

The above uses long options for clarity, in a real life situation you would probably be using short options instead. That means -c instead of –stdout and s instead of –report-identical-files.

$ gunzip -c /var/log/syslog.1.gz | diff -s - /var/log/syslog.1
Files - and /var/log/syslog.1 are identical

 

Bookmarklet to view OSM from coordinates parsed from URL
Aug 12th, 2016 by miki

Here’s a quick Javascript bookmarklet I threw together for opening a new browser window showing an area in OpenStreetMap that is defined from extracting the current window’s URL and searching after useable values for latitude, longitude and zoom.

It grew out of an annoyance over Mapillary’s rendering of the Danish endpoint of the under construction HVDC Cobra Cable(more about it from 4C, even more from EnergiNet and the EU) in Endrup nearby where I live.

In Mapillary’s tiles rendering it is just a bunch of roads and a single POI indication. Whereas the Mapnik rendering shows the existing power infrastructure and the area under construction.

Cobra HVDC, Endrup, Mapillary vs. OpenStreetMap

Well, I have been subjecting my oldstyle C brain to some Javascript lately so I decided to use that haemorrhage for attempting to put together a bookmarklet extracting coordinates from the current window’s URL and opening a new with the same approximate location in OSM.

It ended up like the below code block, and should also be usable on any other sites which receives locations via URL (both using HTTP GET notation with ‘?’ and locally in the page using anchors with ‘#’) and identifying them with key-value pairs using common names ({z,zoom},{lat},{lng,lon}). Note that this doesn’t include OpenStreetMap itself neither Google Maps as they only use the lat/lon values.

javascript:(function (){params={};kvs=document.location.href.split('&');kvs.forEach(function(kv){if(kv.indexOf('?'))kv=kv.substr(kv.indexOf('?')+1);if(kv.indexOf('#'))kv=kv.substr(kv.indexOf('#')+1);skv=kv.split('=');params[skv[0]]=skv[1];});window.open('http://openstreetmap.org/#map='+(params.z?params.z:13)+'/'+(params.lat?params.lat:55.5)+'/'+(params.lng?params.lng:(params.lon?params.lon:8.5)));console.log(params);})();

Copy and paste the above into the “Location” or “URL” of a bookmark and you’ll be able to click it to open a new OSM window on, at least for Mapillary maps pages, the same location as the original site. If nothing is found it will default to coordinates of my hometown of Esbjerg at 55.5/.8.5.

Here’s a prettified edition of the code:

params={};
kvs=document.location.href.split('&');
kvs.forEach(function(kv){
  if(kv.indexOf('?'))
    kv=kv.substr(kv.indexOf('?')+1);
  if(kv.indexOf('#'))
    kv=kv.substr(kv.indexOf('#')+1);
  skv=kv.split('=');
  params[skv[0]]=skv[1];
});
window.open( 'http://openstreetmap.org/#map='
            +(params.z?params.z:(params.zoom?params.zoom:13))+'/'
            +(params.lat?params.lat:55.5)+'/'
            +(params.lng?params.lng:(params.lon?params.lon:8.5))
           );
console.log(params);

Possible TODOs

  • get rid of default location, warn instead
  • support and test more sites (gmaps/osm!)
  • scan page contents for other geo markers
Frostlight RGB LED strip
Aug 1st, 2016 by miki

Stumbled across a very cheap RGB LED strip where I live in Denmark from the brand Frostlight. Priced at down to DKK 50 ~ EUR 6.5 ~ USD 7.5 in Fleggaard at Danish/German border but goes for around DKK 200 ~ EUR 26 ~ USD 30 in the ordinary DIY and internet shops (still cheap compared to other sources). For this amount you get a product which on the packaging is called “3 meter farvede LED bånd (RGB)” (which is a little gibberish Danish and not grammatically correct), English: “3 meter colored LED strip (RGB)”, containing these components;

  • 3 meter strip
  • Controller + IR receiver
  • PSU (Power Supply Unit), 230 V->12 V, 22 W
  • IR Remote control

According to the description this setup “does it all”; RGB multi color LEDS, controller doing colour change, fading etc.

  • Length: 3 meter
  • LEDs: 90 (30 LEDS pr. meter, 3.33 cm between LEDs)
  • Width: 10 mm
  • Colors: 16
  • Burn time: 20.000 hours
  • Silicone protected

The big question for me as a maker/hacker/tinkerer was; Does it use individually addressable LEDs?

And no, it doesn’t;

Frostlight RGB LED, strip segment interconnection

Obviously (as could be expected from the price), this strip is made from plain RGB LEDs with discrete R, G & B LEDS in a common anode setup (12V pin is common, current needs to be sinked from each RGB to control colour and intensity).

The brand Frostlight is unknown to me (they have a very non-informative website without any real product information), but they seem to supply LED products to many discount supermarkets in Denmark. They have a youtube channel (which is not even mentioned on the homepage) containing some product information. Even one for the “Frostlight LED farve-bånd”.

I was looking for a quick and cheap way to source LEDs for the awesome WordClock project by grahamvinyl (Arduino source code at github.com/grahamvinyl/WordClock_color_edit). However, it won’t work as all LEDs on the strip will light up in the same colour, but I consider using it for a cheaper tweak of it.

At least I’m confident I’ll find something to use the strip for anyway.

 

»  Substance:WordPress   »  Style:Ahren Ahimsa
© 2016 Mikini Services