Monday, November 30, 2009

Topic Relevant Comment Spam

On my previous post "Streaming Audio to your iPhone", I received a very odd comment. It was related to the subject, so it didn't immediately look like spam, but it wasn't topical.

Here's the comment (I've removed it from the original post):

Web casting, or broadcasting over the internet, is a media file (audio-video mostly) distributed over the internet using streaming media technology. Streaming implies media played as a continuous stream and received real time by the browser (end user). Streaming technology enables a single content source to be distributed to many simultaneous viewers. Streaming video bandwidth is typically calculated in gigabytes of data transferred. It is important to estimate how many viewers you can reach, for example in a live webcast, given your bandwidth constraints or conversely, if you are expecting a certain audience size, what bandwidth resources you need to deploy.

To estimate how many viewers you can reach during a webcast, consider some parlance:
One viewer: 1 click of a video player button at one location logged on
One viewer hour: 1 viewer connected for 1 hour
100 viewer hours: 100 viewers connected for 1 hour…

Typically webcasts will be offered at different bit rates or quality levels corresponding to different user’s internet connection speeds. Bit rate implies the rate at which bits (basic data units) are transferred. It denotes how much data is transmitted in a given amount of time. (bps / Kbps / Mbps…). Quality improves as more bits are used for each second of the playback. Video of 3000 Kbps will look better than one of say 1000Kbps. This is just like quality of a image is represented in resolution, for video (or audio) it is measured by the bit rate.

It was posted by "Andy". I was ready to post a comment thanking Andy for the additional information, but I decided to look to see if it was copied from somewhere else.

It seems that the exact same comment is placed on just about any blog post that mentions streaming. On the Google search I used, I found a lot of comments from the past couple of months.

Seems "Andy" has been a busy boy, and is actually a dirty, rotten spammer.

"Andy" (blogger), who also posts as "andylock", or "Andy Lock" (Facebook) is an automated spam program for vsworld.com, and the website is a flash-only website. I didn't stay there long enough to really figure out what they were selling, but it looked like some sort of contracting agency in India.

Still, I found it interesting that it wasn't immediately obvious that the post was spam.

Monday, November 23, 2009

Audio Streaming to the iPhone, Take Two

I wasn't completely happy with the results of using M3U files. While it did allow me to specify all of the files, I wasn't able to skip forward and back through the song list.

It turns out that the M3U support is there primarily to support live streams. For example, if you have a live stream from a video source (like a TV tuner card), the server records it, splits it into small chunks and converts it to H.264. The iPhone will then use the M3U file as an index to the individual data files, refreshing the M3U file occasionally. I'll have to set that up next. :)

So I went back to Google to see what I could find. I had come across the <OBJECT> tag in my previous searches, but I had discarded it as too difficult. I didn't want to work in HTML, and object embedding just seemed dirty. However, if I wanted to get access to skip forward/back, it looked like I was going to have to use it. I went through and modified the perl program from yesterday to produce an HTML page for each M3U file. It wasn't until I got to the end that I realized that I had done it wrong.

Lots of things don't work with the OBJECT tag, primarily the GOTO command. It seems that the proper way to embed things into a document is with the EMBED tag. That isn't to say that the EMBED tag is prettier. It isn't, they are both nasty. It is platform and player specific. Luckily, this only has to work on the iPhone/iPod Touch, so I'm lucky that way. I would hate to have to support this for multiple clients.

The first nasty surprise is that while you can have multiple songs in a list, you are limited to 255 of them. Even worse, they aren't done through object references in the rest of the document, the entire list is in the single object tag. That makes it harder to do dynamic, on the fly modification of the list. No small cgi to do shuffles here!

Still, 255 songs is enough to cover pretty much all of my artist directories.

It also seems that Mobile Safari ignores GOTO commands. In regular Safari, you are able to loop back around to the start of the playlist by putting "QTNEXT255=GOTO0" into the embed tag. Looks like Apple doesn't want playing loops on the iPhone.

Next, the iPhone ignores the "autohref", "autoplay" and "autostart" parameters. It always waits for user interaction. This is because the object is not really embedded, it takes full control of the screen. If it did start automatically, it would cause problems on many other sites. It's a small pain, but we'll survive.

I still wish it properly supported M3U files. The seamless transitions are nice. With the QTNEXT, there is a definite pause between tracks.

Here is the updated Perl code:

#!/usr/bin/perl

use File::Find;
use File::Basename;

use vars qw/*name *dir *prune/;

my $URL_BASE= "http://10.10.10.5/";

my @M3Us;

sub createPlaybackHTMLHeader {
    my ($filename, $target, $song) = @_;

    my $title = basename(dirname($filename));

    # open the file, 
    open(HTML, ">$target");

    # Write the HTML/Head elements,

    print HTML <<END;
<html>
  <head>
    <title>$title</title>
    <meta name="viewport" content="width=device-width; initial-scale=1.25"/>
  </head>
  <body>
    <p>Play all music in the "$title" directory, click the button below</p>
    <embed src="$song">
      autoplay="true"
      controller="true"
END
;
    # close the file.
    close(HTML);
}

sub createPlaybackHTMLFooter {
    my ($filename, $target) = @_;

    # open the file, 
    open(HTML, ">>$target");

    # Write the HTML/Head elements,

    print HTML <<END;
      qtnext255="GOTO0"
    </embed>
  </body>
</html>

END
;
    # close the file.
    close(HTML);
}

sub createPlaybackAddSong {
    my ($filename,$target, $song, $count) = @_;

    open(HTML, ">>$target");
    print HTML <<END;
      qtnext$count="<$song> T<myself>"
END
;
    # close the file.
    close(HTML);
}

sub convertM3UToHTML {
    my ($m3ufile, $target) = @_;

    print "Converting $m3ufile to $target...";

    if ( ! -f $m3ufile ) {
 print "No m3ufile, returning\n";
 return;
    }

    # Open that m3u file, and convert it to a playback html page.
    open(M3U, $m3ufile) or die "Can't open m3u file";
    my @songs=<M3U>;
    close(M3U);

    if (@songs > 255) {
 print "Too many songs! Oh well\n";
 @songs = @songs[0..254];
    } elsif (@songs == 0) {
 print "No songs, returning\n";
 return;
    } else {
 print "No problems!\n";
    }

    my $first_song = shift @songs;

    chomp $first_song;

    createPlaybackHTMLHeader($m3ufile, $target, $first_song);
    my $count = 1;

    foreach my $song (@songs) {
 chomp $song;
 createPlaybackAddSong($m3ufile, $target, $song, $count);
 $count += 1;
    }

    createPlaybackHTMLFooter($m3ufile, $target);
}

sub entering {
    print "entering Directory boundary ", $File::Find::name, "\n";

    push @M3Us, $File::Find::dir;

    if ($File::Find::name =~ /.AppleDouble/) {
 return;
    }

    return sort(@_);
}

sub leaving {
    print "leaving Directory boundary ", $File::Find::name, "\n";

    my $directory = pop @M3Us;

    # open the file, 
    my $source = $directory . "/PLAY.m3u";
    my $target = $directory . "/PLAY_ALL.html";

    convertM3UToHTML($source, $target);
}

sub wanted {
   print "Checking ", $File::Find::name,"\n";

   if ( $File::Find::name =~ /\.mp3/ ) {
       print "MP3 Found ", $File::Find::name, "\n";
       my $url = $File::Find::name;
       $url =~ s/ /%20/g;
       $url =~ s/\/export\///g;
       $url = $URL_BASE . $url;
       foreach my $dir (@M3Us) {
    my $m3u_file = $dir . "/PLAY.m3u";
    open (M3U, ">>$m3u_file") or die "Boom!";
    print M3U "$url\n";
    close(M3U);
       }
   }
}

sub cleaner {
    if ($File::Find::name =~ /PLAY.m3u/ ) {
 print "Removing ", $File::Find::name, "\n";
 unlink($File::Find::name);
    }

    if ($File::Find::name =~ /PLAY_ALL.html/ ) {
 print "Removing ", $File::Find::name, "\n";
 unlink($File::Find::name);
    }
}

find ({ wanted => \&cleaner},"/export/mp3");

find ({ wanted => \&wanted , preprocess => \&entering, postprocess => \&leaving},"/export/mp3");

open(M3U, "/export/mp3/PLAY.m3u") or die "Unable to open play";

my @main_m3u = <M3U>;
close(M3U);

my @music = grep(!/AudioBooks/, @main_m3u);
my @random = sort { int(rand(3))-1 } @music;

open(M3U, ">/export/mp3/MUSIC.m3u") or die "Unable to open MUSIC.m3u";

print M3U @music;
close(M3U);

convertM3UToHTML("/export/mp3/MUSIC.m3u", "/export/mp3/MUSIC_PLAY.html");

open(M3U, ">/export/mp3/RANDOM.m3u") or die "Unable to open RANDOM.m3u";
print M3U @random;
close(M3U);

convertM3UToHTML("/export/mp3/RANDOM.m3u", "/export/mp3/RANDOM_PLAY.html");

Sunday, November 22, 2009

Streaming Audio to your iPhone

I'm trying to hide the CDs and DVDs that are steadily taking over my house. I've managed to get rid of all of the DVDs, they've all been ripped to the server and are now happily up in the attic. The last problem are the CDs.

Yes, I have them all ripped to the server. The problem is that I don't have any way of getting at them everywhere in the house. Specifically, my wife likes to listen to murder mysteries as she cooks (me, I like cooking to the news). Since I don't have a network enabled stereo, I have to put up with a stack of CDs sitting on the counter.

I have given up waiting for a network enabled stereo system. The squeezebox just doesn't do it for me. I want something small, with wifi and wired, with a radio, clock and a streaming mp3 player. It should also have .m3u support and a "shuffle".

I decided to see what I could do with what we already have...

  • First realization. We both have iPhones.
  • Second realization. We have a WiFi network in the house.
  • Third realization. The iPhone will happily stream music off of a web page.

So the project was formed. First, I tried the SqueezeBox server (previously SlimServer) a server which serves up streams to network attached stereo component equipment. I've got a 1st generation SliMP3, it was a great device. The server just doesn't play well with the iPhone. So, I decided to go the bare bones route. I decided to setup Apache on my file server so that it will serve up music to the iPhones in the house. Then the phones can be used to stream the music/books wherever anyone is in the house. I would finally be allowed to hide all of the CDs! Perfect!

Setting up Apache was pretty easy. I followed the instructions that are easily Googled. I didn't create any HTML files, but I did configure the auto_index module (/etc/apache2/mods-available/autoindex.conf). My changed settings were:

IndexOptions SuppressDescription SuppressSize SuppressLastModified
IndexOptions SuppressHTMLPreamble
HeaderName /include/iPhone_Header.html

NOTE: HeaderName is relative to the DocumentRoot, not the filesystem.

This allowed me to keep the listing to just the filenames, and replaced the standard HTML header with one of my own:

<html>
<head>
<meta name="viewport" content="width=device-width; initial-scale=1.25"/>
</head>
<body>

This header provides a hint to the iPhone of where to set the viewport. It seems to work for my listings, making the file list usable.

Once we have that, we've got a server with a directory listing that we can scroll through and play individual MP3s. We don't have any playlists though, which is pretty unpleasant.

Bring in the Perl!

I've written a small piece of Perl code to iterate over my MP3 tree and create M3U files in each directory containing URLs for all of the MP3s that are children of that directory. Because of how I store my MP3s, that gives me album, artist and full play lists. I then randomize the playlist to give me a shuffled list.

The new iPhone release is able to play M3U files, with one problem. You can't skip to the next track, which is pretty poor. But now I've got a method of delivering music to any room in the house. I just have to get one of those 3rd party iPod speaker systems, and I can get rid of all of the CDs in the kitchen!

#!/usr/bin/perl

use File::Find;

use vars qw/*name *dir *prune/;

my $URL_BASE= "http://10.10.10.5/";

my @M3Us;

sub entering {
    print "entering Directory boundary ", $File::Find::name, "\n";

    push @M3Us, $File::Find::dir;
    return sort(@_);
}

sub leaving {
    print "leaving Directory boundary ", $File::Find::name, "\n";

    pop @M3Us;
}

sub wanted {
   print "Checking ", $File::Find::name,"\n";

   if ( $File::Find::name =~ /\.mp3/ ) {
       print "MP3 Found ", $File::Find::name, "\n";
       my $url = $File::Find::name;
       $url =~ s/ /%20/g;
       $url =~ s/\/export\///g;
       $url = $URL_BASE . $url;
       foreach my $dir (@M3Us) {
    my $m3u_file = $dir . "/PLAY.m3u";
    open (M3U, ">>$m3u_file") or die "Boom!";
    print M3U "$url\n";
    close(M3U);
       }
   }
}

sub cleaner {
    if ($File::Find::name =~ /PLAY.m3u/ ) {
 print "Removing ", $File::Find::name, "\n";
 unlink($File::Find::name);
    }
}

find ({ wanted => \&cleaner},"/export/mp3");

find ({ wanted => \&wanted , preprocess => \&entering, postprocess => \&leaving},"/export/mp3");

open(M3U, "/export/mp3/PLAY.m3u") or die "Unable to open play";

my @main_m3u = <M3U>;
close(M3U);

my @music = grep(!/AudioBooks/, @main_m3u);
my @random = sort { int(rand(3))-1 } @music;

open(M3U, ">/export/mp3/MUSIC.m3u") or die "Unable to open MUSIC.m3u";
print M3U "@music";
close(M3U);

open(M3U, ">/export/mp3/RANDOM.m3u") or die "Unable to open RANDOM.m3u";
print M3U "@random";
close(M3U);

Sunday, November 15, 2009

Oracle's BETWEEN keyword

I came across a new Oracle keyword today, BETWEEN.

At first, I thought it was pretty cool. I would be able to simplify the majority of the range checks that I perform. Before I really started using it though, I decided to look at what it actually did. Ouch!

Google, my ever present documentation source, told me that between doesn't work the way I thought it would. It's inclusive of both ends of the range. Who would want that? You never want a range that is inclusive of both ends! Otherwise, elements in contiguous ranges have indeterminate ownership!

Let's try an example:

We want all rows with a date on the row that has a date of today.

  select id from test_table where my_date between 
    trunc(sysdate) and trunc(sysdate) + 1;

Great you would think, almost too easy!

You would be correct too, it was too easy. It doesn't work. Since it is inclusive of the end of range, you get all values that have a my_date of 12:00:00 tomorrow. What you really want is:

  select id from test_table where 
      (my_date >= trunc(sysdate)) and 
      (my_date < trunc(sysdate) + 1);

When using new keywords and abstractions, you should always know what they are doing.

Monday, November 02, 2009

std::auto_ptr and GDB.

I needed to gain access to the contents of an auto_ptr inside of GDB. However, GDB doesn't like the overloaded -> operator, so the simple foo->fnImInterestedIn() doesn't work. Here's the simple pattern:

class bar_t {
  public:
     int fnImInterestedIn();
};

std::auto_ptr<bar_t> foo;
(gdb) p ((struct bar_t *)foo._M_ptr)->fnImInterestedIn()

Hulu Proxy Apocalypse

It seems that the great Hulu apocalypse has hit more than just Witopia.net, Amazon's EC2 instances also seem to be blocked.

Of course, there are a tonne of other cloud providers out there, several of them even cheaper than Amazon. Personally, I'm still with Witopia. Like any good company, they had a new address range up immediately.

Hulu is stuck in the same game as Apple. They're playing whack-a-mole with the hackers. The only problem, every time they want to block an access method, it costs them money. However, it is free for us to invent a way around a block (it's a hobby), and there are a lot more of us than there are developers at Hulu.

This is yet another example of "Don't piss off the nerds".

Personally, here are the lessons that I would take from this.

  1. There is a market for International access. People are willing to _pay_ to get access. I am currently paying US$12/month to access Hulu, Pandora, and TV.com. I would pay that for unfettered access to Hulu.
  2. Hulu isn't going to win with black lists. They're going to have to implement a white list, which is a lot more expensive to maintain.
  3. Not a single person who was using Hulu simply stopped watching TV when Witopia and Amazon were turned off, they just went back to bittorrent.
  4. Don't piss off the nerds. They can out-spend you.

There are plenty of cloud providers out there. The same solution will probably work on other sites. Install Squid and give it a try!