Spam-tagging russian encoded spam with sieve rules and its regex extension

Since a few weeks I have been receiving numerous e-mails in russian language. Sadly enough, I don’t speak Russian. However, about 20 or 30 mails per day made it through my three layers of spam detection. Two Spam Assassins, one at my provider (powered by the experience of several hundred domains), one on my own Cyrus IMAP server including my preferences and “personal experiences” plus (third level) a rather complex set of Sieve mail filtering rules could not prevent those mails from traveling to my Inbox.

From: “?koi8-r…”

Thus I had a look in the mails’ source code and found the “From:”-field starting with “?koi8-r?” like this:

From: =?koi8-r?B?18/azcXdxc7JxSDu5PM=?= <pinstripev6@rodswan.com>

That’s perfectly enough for my Third Stage (no Boston here!). I just had to add the fifth line to my sieve script below – and now now more illegible Russian mails appear in my inbox. Okay, they now are in the “spam”-folder, but lateron I will just redirect the “fileinto”-command and have them rejected.

The Sieve part: 

require [ “fileinto”, “regex”, “reject” ];

[…]

if anyof (

 header :regex :comparator “i;ascii-casemap” [“Subject”, “From”, “To”, “Cc”, “Bcc”] “[.]*[vV]iagra[.]*”,

 header :regex :comparator “i;ascii-casemap” [“From”] “[.]*koi8-r[.]*”,

 header :contains [“X-Spam-Level”] “*****”

)

{

        fileinto “INBOX.spam” ;

        stop;

}

[…]

Howto convert MPC, Flac and Ogg to mp3 on Linux with free tools

In my MP3 library’s past I had experimented with different file formats. I have recordings of vinyl or radio in FLAC (Free Lossless Audio Codec), MPC (Musepack), Ogg (Ogg Vorbis) and other, extracted audio tracks from DVB-recordings on my PC. But since my car’s radio, my phone and my DVD player don’t support those formats (Nokias N900 did!), I chose to convert them to MP3. I made three little bash scripts, that probably only prove my limited coding capabilities… 🙂

Nevertheless, these little snippets are quite handy for me, and yes, I do know there are better solutions around, but that simply works for me. For security reasons, the source files are not deleted, but moved to a new subdirectory. The final “ls” creates a playlist in the working directory.

If you want to try that at home, you’ll need the programs:

– oggdec from the vorbis-tools package

– flac from the flac package

– mpcdec from the package musepack-tools

All of them are in standard Ubuntu repositories and can easily be installed via aptitude (at least here on Maverick (10.10).

a) FLAC, via stdout:

#!/bin/bash
for file in *.flac
do flac -cd “$file” | lame -h – “${file%.flac}.mp3”;
done
mkdir flac.old
mv *.flac flac.old/
ls *3 > playlist.m3u

b) … and the same for MPC:

#!/bin/bash
for file in *.mpc
do mpcdec “$file” – | lame -h – “${file%.mpc}.mp3”;
done
mkdir mpc.old
mv *.mpc mpc.old/
ls *3 > playlist.m3u

c) Ogg Vorbis:

Directing Oggdec to stdout wouldn’t work here, so I needed a temp.wav file for each conversion process:

#!/bin/bash
for file in *.ogg
do oggdec -o temp.wav “$file”;
lame -h -V4 temp.wav “${file%.ogg}.mp3”;
done
rm temp.wav
mkdir ogg.old
mv *.ogg ogg.old/
ls *3 > playlist.m3u

Next thing will be extracting the audio track from 3GP files, like ancient mobile phones used to record. As far as i read up to now, Mplayer or its partner mencoder should be able to do that, e.g. for videos with

mencoder infile.3gp -oac pcm -ovc raw -o outfile.avi

I’ll keep you updated.

Flash, web videos and free Software with Mediathek and JWplayer 5.3

I am just testing two (not only at first sight) promising video tools:

Whereas Jwplayer is a javascript-based player for websites that streams FLV, H.264 and other videofiles to a client with a flash-enabled browser, Mediathek is a Java-GUI that works as a frontend for many (really many) online video stores like those of german TV stations. (Update: and supports also podcasts and works as a digital TV recorder based on downloading and subscribing to streams.)

JWplayer

JWPlayer is a software that many big sites are using, like the infamous kino.to, and it seems very robust. Though it’s open source, there is a (not really expensive) commercial version available for some bucks. Nothing much to say, it just worked here.

Mediathek

Mediathek comes as a .jar archive, I tried version 2.3.0 that is marked as a "test" version, but it simply worked. Download and extract the .zip archive and enter "java -jar Mediathek.jar" in the subdirectory Mediathek_2.3.0, and you get the blank start screen after an automatic check for updates.

The very first step for everyone will be to update the film list via the prominent button "-> alle Filme neu laden" (What’s that in the english version?) and shortly after there will be a long list full of free available videos (see the screen shot below). Nice.

As you can see, Mediathek not only allows searching, viewing and listing of all those free videos, it also has three buttons to record them to file. That’s great for everyone who wants to create his own archive of his favorite TV program.

Please note: For instant replay the Video Lan Client Vlc must be installed, for recordings either Mplayer or Flvstreamer. Vlc is a great recommendation also for all Windows users, it really rocks. It checks and updates codecs automatically, doesn’t install nasty stuff that isn’t really needed and just works without consuming much ressources. Forget all the other spyware like Bill’s Mediaplayer, give it a try. Here you can see VLC on Ubuntu 10.10:

And yes, Mediathek needs a current Java version – on my Ubuntu Maverick (10.10) :

# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.1) (6b20-1.9.1-1ubuntu3)
OpenJDK 64-Bit Server VM (build 17.0-b16, mixed mode)
#

worked fine.

Update: Subscriptions and Podcasts, minor issues with paths

After some testing, I found out – to my big surprise – that Mediathek can also handle podcasts and supports so-called subscriptions (in german: Abonnements), where it tries to automatically detect new versions of TV programs available and lets you download and save them to disk at a mouse click. However, the default settings are not quite clear.

First: Subscriptions and podcasts have to be enabled in the main menu, only then the two separate tabs will appear – that’s not too obvious…

Second: You should set the path of downloaded/recorded files to some reasonable value. Its default points to /tmp, which might make sense on many systems. But as the dialog for new subscriptions and podcast downloads also offers a path entry, this is not really clear. Mediathek saves the file to the path: "Target_path_from_global_[abo|podcast]_settings / individual_path_from_single_download_settings".

So if Mediathek happens to fill up the wrong partition, have a look in the settings menu for subscriptions (Einstellungen | Abos einrichten) or podcasts (Einstellungen | Podcasts einrichten). In both Dialogs you’ll find a target path (Zielpfad or Basis-Zielordner) variable which I set to my /temp partition – which is completly different from /tmp which is being emptied at every boot.

The same applies to the default recording command that is linked with the buttons in the main window – you can change these programs (Mplayer, Flvstreamer) via the menu entry "Programme | Button einrichten". I found that not so self-evident. But what’s really cool is the fact that you can easily add your own set of programs for such buttons, that is really nice.

Creating your own Webradio “app” for the N900 on Maemo5

Just came across this “App” here that enables your N900 to play the FM4 Webradio stream via a tiny little icon on the smartphone’s desktop. Like most apps, there is really not much magic involved.

I had a look into the Debian package used to install this austrian radiostation‘s App. If you have the extras-devel and/or -testing repositories enabled on your phone (either via Program Manager or by editing sources.list), then an apt-cache search webradio will give you this:

mfeilner@laptop:~$ ssh root@n900

root@n900’s password:

Nokia-N900:~# apts webradio

webradio-fm4 – Startup Webradio fm4.orf.at

webradio-radio-wien – Startup Webradio http://wien.orf.at/

webradio-superfly – Vienna’s Soulful Radiostation (Web Radio)

Nokia-N900:~#

As you can see, I did’nt even have to worry about in which pocket or trouser I left the phone yesterday evening… Back home, it automatically connects to my WLAN, and via OpenSSH I can connect. I added bash and some aliases to the “~.bashrc“, just like I am used to on a PC. Apts is my alias for apt-cache search, and a simple apt-get install webradio-fm4 adds those files to the phone:

Nokia-N900:~# dpkg -L webradio-fm4

/.

/usr

/usr/share

/usr/share/doc

/usr/share/doc/webradio-fm4

/usr/share/doc/webradio-fm4/changelog.gz

/usr/share/doc/webradio-fm4/copyright

/usr/share/applications

/usr/share/applications/hildon

/usr/share/applications/hildon/webradio-fm4.desktop

/usr/share/icons

/usr/share/icons/hicolor

/usr/share/icons/hicolor/48×48

/usr/share/icons/hicolor/48×48/hildon

/usr/share/icons/hicolor/48×48/hildon/wr-fm4.png

Nokia-N900:~#

Yes, the full set of dpkg and apt is available on this phone. Once you had a look into these files, adding another Webradio as an Icon to your Desktop – or as the Mac and Android people would call it – an “App“, is pretty straightforward: Just copy, rename and edit /usr/share/applications/hildon/webradio-fm4.desktop and adapt it to your needs:

Nokia-N900:~# cat /usr/share/applications/hildon/webradio-fm4.desktop

[Desktop Entry]

Encoding=UTF-8

Version=1.0

Name=FM4

Comment=FM4

#Exec=sh /home/user/webradio.sh fm4

Exec=/usr/bin/dbus-send –print-reply –dest=com.nokia.mediaplayer /com/nokia/mediaplayer com.nokia.mediaplayer.mime_open string:”http://mp3stream1.apasf.apa.at:8000/listen.pls&#8221;

Icon=wr-fm4

Terminal=false

Type=Application

X-HildonDesk-ShowInToolbar=true

X-Osso-Type=application/x-executableNokia-N900:~# vi /usr/share/applications/hildon/webradio-fm4.desktop

You only need to change the URL of the webcast in the line “Exec” and probably you would want to change the name in “Name” and “Comment”. I added another Icon in /usr/share/icons/hicolor/48×48/hildon/ and pointed the line starting with “Icon” to that file. Opening, Copying and Editing the icon image (a 48×48 pixel sized PNG) works fine with Gimp, since the N900’s SSH-Server fully supports the FISH-Protocol integrated in Konqueror and Dolphin. The app now is available in the programm menu and as a widget for the desktop.

My next step will be finding out about recording music from services like shoutcast and last.fm or video recording from flash sources with the N900. The 32 GByte of internal memory must be filled somehow :-). By the way: there were already about 100 webradiostations preconfigured with my device, and Mplayer is available – so the recording shouldn’t be that difficult. And flash works fine, even webtv works – if you keep the resolution down. Hail to all the apple fans, and thank god there’s no need for I-tunes in this world – except for dummies.

 

LXDE rocks … 35 Seconds for a reboot…

I am really amazed. Klaus Knopper’s Knoppix 6.3 (still exclusively available with the German Linux Magazine) brought me to LXDE, the smart and fast Desktop environment. Together with Ubuntu (still 9.10, but soon 10.4!) my laptop, a Intel Dual Core 1.4 Mhz with a SSD needs only 35 Seconds for a complete reboot from and to the graphical environment. And even my old Desktop-PC (an AMD Athlon 3000 with a prekarbonic IDE disk) manages that in less than 90 seconds  for a complete reboot. Wow.

Who needs suspend these days? And all you Windows freaks, digg this.

Getting MMS, USSD Codes and HD movies to work on Nokias N900

This N900 from Nokia is really a great thing and full of stunning technology. Nevertheless it’s not fully functional for me, at least the way it comes to the customer. I just got back my test device from an external author, who flashed it back to vendor defaults. This gave me the chance to find out how things have changed with Maemo 5, the Ovi Store and the newer updates of Nokia’s Debian operating system.

What I am still missing is support for MMS, USSD Codes and some extensions for movies, even though there is obviously no better supporta around in a mobile phone than there is in the N900, except for portable DVD-Players. Just behold: Apple and Palm can’t do Flash, we are talking about HD and DivX here!

Ok, here’s my wishlist: I have O2’s multicard enabled, I want to send MMS to my beloved wife when I am on the road, and I want to watch my HD-TV recordings on my mobile phone and in Hotels via the TV-out of the N900

So here is what I did:

First thing was adding some repositories: The following is my “sources.list”, or to be precise, my “hildon-application-manager.list” in “/etc/apt/sources.list.d”:

Nokia-N900-02-8:/etc/apt/sources.list.d# cat hildon-application-manager.list
deb https://downloads.maemo.nokia.com/fremantle/ssu/apps/ ./
deb https://downloads.maemo.nokia.com/fremantle/ssu/mr0 ./
deb https://downloads.maemo.nokia.com/fremantle/ovi/ ./
deb http://repository.maemo.org/extras/ fremantle free non-free
deb http://repository.maemo.org/extras-testing/ fremantle free non-free
deb http://repository.maemo.org/extras-devel/ fremantle free non-free
Nokia-N900-02-8:/etc/apt/sources.list.d#

Take care, because the “testing” and  “devel” packages may break your system. If you need a working phone, leave out the last two lines and wait until the MMS function is built in by Nokia. If you prefer the hard way, by using the click and click touchscreen-app-manager, read this. Oh, for all of this I quickly installed SSH-Server, SSH-Client, Midnight Commander and Wget, all through “apt-get install” commands, , like on any other Debian PC.

Then I followed the instructions for the Unstructured Supplementary Service Data (USSD) Pad I found here (Thanks Pycage!) and I got a USSD Icon in the program manager. Now switching from one sim card to the other with *123# works, and with *125#  I can control which of my mobile phones shall receive SMS, MMS or calls. Obviously there’s more things that can be done with that:  Owners of prepaid sim cards can check their balance with *100#, a full list of codes is here and here. Both links are in German and for German providers. Your mileage may vary… 🙂

Next step: MMS. On the symbian60 website there is a nice howto that provides an easy way how to enable MMS on your N900. In brief, you need two apps and a little configuration work, including adding a new internet (WAP) access point. After that you have an extra app, not yet integrated with your conversations, but with your contacts. You can send and recieve MMS and choose contacts from your adressbook. Resizing an image from 5 megapixel size to 300-px-MMS-size may take quite a while on the N900’s ARM-600-Mhz-CPU, but it works. And it does nicely.

Don’t forget to go online with the new MMS profile before sending! Took me some time to figure that out… 🙂

Finally: Getting OGG and  Divx-encoded High-Resolution Movies to work on the N900. That’s even simpler, since there are a lot of Media Players out there, and there are codec-packages for your Debian-Maemo-combination available . However choosing the right movie may be difficult due to the restricted hardware. Don’t force your device to do scaling – that won’t work in most cases. If you want to make sure your N900 runs with clean and correct movie files, go to the Internet Archive and download some HD, Ogg, divx, H.264 movies or other files. These movies are freely available, a really great library. Download them to “/home/user/MyDocs/.videos/” on your mobile, and the media player will show them. Or – for the fans of animation, go and get the open source films “Elephants Dream” or “Big Buck Bunny“- all rendered with Blender. This screen shot shows my wget process on the N900 downloading Elephants dream to the mobile.

But even though Kmplayer and others can play also 1920-HD-Videos, watching them won’t be fun – the hardware is too slow. Big buck bunny in 1080 Ogg vorbis plays, but is no fun either. From my experience you better make sure that your videos’ resolution is less than or equal to DVD (about 800×600 should work), and then you can have brilliant movies on the Beamer or TV via the built-in cinch connection.

One thing must be added: There are so nice shortcuts (more tipps and tricks here) on the N900: Have you tried portrait mode in the browser? Hit CTRL+Shift+P to activate it, then turn  your phone around. For screenshots on the N900, hit CTRL+Shift+P and go to “/home/user/MyDocs/.images/screenshots/”.

And if you want a wallpaper series on your desktop – like putting a panorama photo from your  holiday  there? Just have a look at the “.desktop” files in “/home/user/MyDocs/.images/”, copy 4 images there and edit on of these files to suit your needs. Nice, isn’t it? I did this with vi on the N900. Works perfectly, I am sure that also Emacs can do that.

OpenVPN 2.1 rc20 is out

For those who haven’t noticed yet: A new release candidate of OpenVPN 2.1 is out. Its number is 20, the changes are mostly bug fixes. Here is the download.

Pidgin with Facebook Chat

There are some extensions/plugins available, that enable Pidgin to work together with Facebook chat. I am still trying, but this seems to work. Here is the Suse repo (thanks to Justin Haygood) , and on the project’s page you’ll find source, binaries and debian
Next thing I’ll try is skype for pidgin… wait…. does this make sense?

🙂

Kontact Journals: KDE’s Blog Engine

Again I found something I really like. Kontact can handle a blog as an ordinary calendar, well in fact as a “journal”. Of course this is not as powerful as Kblogger or other tools, but just the right size for a convenient blog entry through the day. Got a link via E-Mail (K-Mail) or a RSS-News via Akregator that you want to post?

Just add a Blog Ressource to your Journal (either in Kontact or on the KDE-Ressources Tab of Kcontrol), enter your credentials and start blogging. You can even manage and edit your old entries, kategories and more.

I will surely blog more on this, but up to now I am still testing the software. This is running on Suse 11.0 with KDE 4.3.2. At first glance I see that buttons for images and hyperlinks are missing… but the tool works on three machines now… 🙂

Change Wine language from command line

Since I am working for German and English Newspapers alike, I often need to do screen shots of programs in different languages. On Linux no problem, but on some Windows systems maybe even a violation of licence or law… So for MS screenshots I tried the new Wine version 1.0 on Linux – that seems to run great – if you’re not a gamer. You call the program simply with the language code you need:

LANG=en_US wine .wine/drive_c/Programme/xca/xca.exe

for example starts Xca in English, while:

LANG=de_DE wine .wine/drive_c/Programme/xca/xca.exe

enables you to make German screenshots. Great. This works also with Japanese, Chinese or whatever language available.