Adding value?

Ask yourself, "Can I be replaced with a shell script?" Specifically:

 % yes 'No'

Then ask yourself, "What value am I adding?"

Google Chrome, a Contrarian View.

I read the comic about Google Chrome. I love the architecture! We downloaded it here in the office, and sure enough - it forks a new process for every page load. This is the browser architecture that I've been waiting years for, one where a script on one page won't kill performance in another window.

So, I went home and tried it on my single core Athlon at home. Ouch! The performance is terrible. The browser keeps locking up, it stops responding to mouse clicks, keypresses, anything and everything. It acts like alpha quality software, not the "beta" it is supposed to be.

I've got a 2 year old system (Athlon64 3800+), the only difference between it and new ones is that it is single core. It's still fast enough to play new games on, but it can't handle Google Chrome.

I can only assume that Google Chrome has a tendency to starve some of their child processes on single core machines.

In the meantime, I'll stick with Firefox 3. It leaves Google Chrome in the dust.

Duck Typing Sucks.

I'm working with some code that's written in Python, and makes extensive use of duck typing. In duck typing the premise is that if it quacks like a duck, it is a duck.

Of course, if you've got code that doesn't tell you what it expects to receive, you can't hope to figure out what the type should be.

For example, let's say you've got 3 code blocks:

class A:
    def funkyDuck():
        # random cool stuff

class B:
    def funkyDuck():
        # entirely different cool stuff

class C:
    def funkyDuck():
        # a third entirely cool thing

Now, imagine you're looking at code that goes:

some_random_object.funkyDuck()

Which one gets called? By the way, they're all in different files, so the only way to find them is with "grep". Added bonus, the undocumented hash!

    def addApplications(self, appDict):
        """ Add some application modules to the defaults.
            All binaries and libraries in any of the source dirs will be soft linked
            to a directory with the name of the product.
            @type  appDict: dict of string -> (list, string)
            @param appDict: The source dirs and OS user name for each product
        """

Guess I'm supposed to guess what appDict actually looks like.

Hrm. Perhaps my problem isn't with duck typing itself, but it's use here.

Hint: When your test framework is just as complicated as the program under test, you're doing something very, very wrong.

One in a Million

I remember discussing multi-threaded programming with a senior engineer the better part of a decade ago. I was reacting rather strongly to his cavalier attitude towards multi-threaded code, deadlocks and race conditions.

"That is such a small hole, it's one in a million that it will happen."

At the time, we were a small company, and I was a young engineer. One in a million, that doesn't sound that big. Of course, one in a million happens more frequently than you would expect. It didn't take very long for our systems to be processing a million transactions a day, and then millions per hour. Every time the holes had to get smaller and smaller.

They still have a fault in there where the entire system crashes when the system goes completely idle for a couple of minutes. It doesn't happen all the time, it's a "one in about 20 million", but it only has to happen once. So, it crashed about once a week. That one took forever to track down. Still haven't been able to fix it entirely, but they know that it's there, and the hole is smaller, probably "1 in a couple billion" now, below the rate that it's restarted for maintenance. Fixed, for now.

It's interesting see that the DNS engineers are learning the same lesson. Compute power has now shifted in favour of the attackers. As we saw with Captcha, it doesn't matter that you only get through rarely - you only have to get through once in a while. Even just guessing a DNS TXID (2^16), you've got a 1 in a 65536 chance of guessing the right answer.

In pure mathematical terms, the likelihood of getting the TXID wrong 400,000 times in a row is 0.2%. In other words, pretty certain.

(1-(1/2^16)))^400000)

Let's look at the 50/50 point, where the likelihood of seeing X number of failures is 0.5.

That turns out to be somewhere around 45000 attempts, simple for loop range. Since I am spoofing packets, I don't care about a response, even better, filling up the pipe will expand the race situation, giving me a larger chance to get through.

Still, it sounds like the attacker has to also be requesting lookups to poison the server. Let's say they take 2ms each, and they're being nice by only doing them one at a time. Even using the large 400k count, it will take at most 15 minutes of concerted effort to poison the cache. The 50/50 point? One minute, 20 seconds.

Look out for those one in a million situations, they happen more often than you'd like.

Naming Documents

I thought we killed off Hungarian Notation because it doesn't work? Did I miss a memo somewhere? It seems that Hungarian Notation is making a comeback, this time in document names.

There's nothing like having your meta data encoded in the file name, with three letter acronyms for everything. It is impossible to decode, and becomes really fun when the team names change once a year!

"Say where's the SRS?" "Is it in SRS.doc?" "Nope" "How about esgEng_SIT_DR4_SRS.doc?" "Nope" "Oh, wait, Engineering was renamed last year for 6 months.... How about esg_PEN_SIT_DR3_SRS.doc?" "Ah, that's got it."

Encoding meta data in the filename is stupid. How about putting it in the file in the meta data portion where it belongs, and then using a search tool to search it? Or, maybe a directory structure that represents your tag tree.

esgEng/SIT/DR4/SRS.doc

Then, when you need to change the meta data, you rename the tree:

esg/PEN/SIT/DR3/SRS.doc

But Jason, I need to know who produced the document without opening it!

Use the checksum, and search/store that. Anything else can be mistakenly altered or lost. Use the file's checksum. It is a much more reliable descriptor of the file than the filename! I have learned from experience to, never, ever trust a filename, they lie.

Or better yet, you buy a document repository off the shelf for 100k, and shove the problem at them. Personally, I just use the Google Search Appliance. Now that they've got it looking at the correct bits of data, it's very useful. Much better than any of the searches built into the various corporate portals.

Getting Starcraft working on OSX

I've started playing Starcraft again after ages away, but I found that my brand spanking new MacBook Pro wouldn't work!

The new versions of OSX have an updated video driver which does not support 256 colour modes. This means that Starcraft, Diablo II, or basically any older game will no longer work. Not good when the applications were still sold by Apple as OSX compatible games (they've since been removed).

I have now figured out a simple (albeit more expensive - US$79.99 + Windows license) way to solve the problem. I decided to run the game under VMWare Fusion.

However there were two problems with this. First, when running in full screen mode, the display doesn't stretch to fill the screen. So on my laptop, I end up with a postage stamp display right in the middle of the screen. Second, the mouse is not restricted to that little postage stamp, it moves freely all the way around the display. When attempting to scroll the display in the game, you move the mouse to the edge, since the mouse will leave the bounds, the game doesn't scroll very well.

The first was fixed with a quick google.

In the file "Preferences/VMWare Fusion/preferences" add

    pref.autoFitFullScreen = "fitHostToGuest"

This will stretch the display. If you have a widescreen display, you will still have black bars on the left and the right, but we're making progress.

Now to mouse capture. There doesn't appear to be an option to prevent the mouse from being taken back by the host OS, so I took a more drastic approach. I uninstalled VMWare Tools. This requires a reboot of the VM, but when you are done, you will no longer be able to move the mouse outside of the VM!

To get back to the host OS, press CTRL-CMD. To re-install VMWare Toole, select "Install VMWare Tools" under "Virtual Machine"

You should now be able to play Diablo II and Starcraft in all their 256 colour beauty.

Now, if I can only control my nerves enough to work my trackball....

Keeping Efficiency Gains

Air Canada has implemented self check-in at Pearson Airport in Toronto. For all flights, you no longer go to an attendant to obtain you boarding pass, you enter all the required data into a computer, and then check your bags.

This was supposedly done for efficiency reasons. However, most of the efficiencies in the system have been lost. Amazingly, when you get to the front of the second line (to check in your bags), the check in clerk does exactly the same amount of work as before. The first thing they do is verify all of the details that you provided the computer!

Let's go over that again. After having provided my information to a computer, I have to provide it a second time to a human who verifies the first data. The time taken for verification was exactly the same as it would have taken for them to key it in in the first time. Instantly, you have a net loss of efficiency. Even worse, they only have a single set of scales for each pair of desks. That means that there is substantial dead time while you wait for the person at the desk next to you to finish weighing their bags. More lost efficiency.

I can see how it happened. Someone checked in to the wrong flight, or their bags went to the wrong place. Perhaps they got to US customs (you go through US customs in Canada when flying to the US), and didn't have the correct forms or all their data provided, and were sent back, maybe they even missed a flight.

So, the check-in clerks, who are also looking to protect their jobs, add in the task of checking customer provided data.

However, they don't check the data of people who don't have checked baggage, they go straight through to customs. This shows the stupidity of the additional check. Is Air Canada saying that people with checked luggage are more likely to enter incorrect data? I doubt it.

If you make efficiency gains in your organisation, make sure you protect them. Guard them jealously. If you don't, you will see them frittered away.

The Truth about Vacations.

This invariably happens to me. I go on vacation, and then the server goes down. It doesn't go down during the year. It's had an uptime of over a year! However, it's down and stayed down. So, if you've been trying to email me, I apologize, it won't get through. The same goes if you're wondering where blog.jason.pollock.ca went to, or why I'm not on jabber anymore. I'm trying to get it going again, but email will probably be down until I get back home. The eBook Repositories will definitely not be updating, so the Baen Books will slowly rot.

Installer.app and chmod/chown

How is it that I would go about chmodding the directories created by CopyPath?

It seems that if the .zip contains the permissions already, they are maintained when used by CopyPath.

In other words, if you have:

foo/bar

in your .zip, and do a CopyPath("foo", "~/foo") in the XML document, the permissions are set up correctly. At least I haven't had any problems if my installer looks like that.

However, if you create implicitly create directories by using CopyPath on the individual files, they are created without permissions, and a chmod/chown is needed. For example, CopyPath("foo/bar", "nuts/baz/bar") will create "nuts/baz", with no permissions.

I have to do this with the Gutenberg SciFi and Baen Books libraries, since I don't host the .zips themselves.

To manage the chmod/chown problem, the packages ensure that the BSD subsystem is installed (or Cydia), and then runs them as part of the install steps.

Here is an example:

      <key>preflight</key>
      <array>
        <array>
          <string>IfNot</string>
          <array>
              <array>
                <string>InstalledPackage</string>
                <string>com.natetrue.iphone.iphone_binkit</string>
              </array>
          </array>
          <array>
              <array>
                <string>AbortOperation</string>
                <string>You must install the "BSD Subsystem" package from Installer first!  It's in the System Category.</string>
              </array>
          </array>
        </array>
      </array>

Which checks for the BSD subsystem as part of the pre-install. If the BSD subsystem is not installed, the user gets a pop-up error message and the package will not install.

And then the install

      <key>install</key>
      <array>
        <array>
          <string>CopyPath</string>
          <string>1013.txt</string>
          <string>~/Media/EBooks/The_First_Men_in_the_Moon/1013.txt</string>
        </array>
        <array>
          <string>If</string>
          <array>
              <array>
                <string>FirmwareVersionIs</string>
                <array>
                  <string>1.1.3</string>
                  <string>1.1.4</string>
                </array>
              </array>
          </array>
          <array>
              <array>
                <string>SetStatus</string>
                <string>Changing Permissions</string>
              </array>                              
              <array>
                <string>Exec</string>
                <string>/bin/chmod -R 755 /var/mobile/Media/EBooks</string>
              </array>
              <array>
                <string>Exec</string>
                <string>/usr/bin/chown -R mobile /var/mobile/Media/EBooks/.</string>
              </array>
              <array>                 
                <string>SetStatus</string>
                <string>Running GutenMark</string>
              </array>
              <array>
                <string>Exec</string>
                <string>/usr/bin/GutenMark --profile=en --no-toc --config=/etc/GutenMark.cfg /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/1013.txt /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/1013-h.htm</string>
              </array>
              <array>
                <string>SetStatus</string>
                <string>Running GutenSplit</string>
              </array>
              <array>
                <string>Exec</string>
                <string>/usr/bin/GutenSplit --no-toc /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/1013-h.htm /var/mobile/Media/EBooks/The_First_Men_in_the_Moon/Chapter_</string>
              </array>
          </array>
        </array>

That's not the full install segment, but it should give you an idea. I have to separate 1.1.3/4 from earlier firmwares because they use different users and store files differently.

To help, I've created a little perl library that allows me to write packages without having to mess with the nasty XML. It's part of the iphoneebookrepo project over at google code. Specifically PlistMaker.pm.

In terms of the perl library, it ends up looking like this:

my $gutenberg_html_install =
 p_if(IsFirmwareVersion("1.1.3", "1.1.4"),
      SetStatus("Changing Permissions").
      Exec("/bin/chmod -R 755 /var/mobile/Media/EBooks").
      Exec("/usr/bin/chown -R mobile /var/mobile/Media/EBooks/.").
      SetStatus("Running GutenSplit").
      Exec("/usr/bin/GutenSplit -1 -2 -3 -4 --no-toc --no-skip /var/mobile/Media/EBooks/$BookDir/${BookNumber}-h.htm /var/mobile/Media/EBooks/$BookDir/Chapter_")
 ).
 p_if(IsFirmwareVersion("1.0.0", "1.0.1", "1.0.2", "1.1.1", "1.1.2" ),
      SetStatus("Changing Permissions").
      Exec("/bin/chmod -R 755 /var/root/Media/EBooks") .
      SetStatus("Running GutenSplit").
      Exec("/usr/bin/GutenSplit -1 -2 -3 -4 --no-toc --no-skip /var/root/Media/EBooks/$BookDir/${BookNumber}-h.htm /var/root/Media/EBooks/$BookDir/Chapter_")
 );

my %BookDescriptor = (
 bundleIdentifier => "ca.pollock.gutenberg.$BookNumber",
 name => $BookTitle,
 version => "1.0",
 location => $BookURL,
 category => "Gutenberg SciFi (" .$BookLang . ")",
 size => $BookSize,
 hash => $BookMD5,
 url => "http://jason-pollock.blogspot.com/",
 maintainer => "Jason Pollock",
 contact => 'jason@pollock.ca',
 description => "by $BookAuthor",
 install => $Book_install,
 preflight => $Book_preflight,
 postflight => "" ,
 uninstall => $Book_uninstall,
 update => ""
 );

my $BookPLIST = CreatePackage(%BookDescriptor);

One "Gotcha" to be aware of... The Exec command is actually performing it's own parameter splitting prior to calling "Exec". It does not do any sort of "~" expansion, nor does it understand spaces. So, if you are doing a chmod on a directory with spaces in the name, it won't work, regardless of escapes or quotes.

The iPhone Gutenberg Science Fiction eBook Repository!

Yes, I managed to finish it. I've managed to convince Project Gutenberg books to install onto the iPhone.

To make use of the repository:

1. Obtain an iPhone.
2. Jailbreak the iPhone, I recommend ziphone for this task.
3. Install Books, the iPhone ebook reader using the Installer application (1.4+ only!)
4. Add the repository to Installer:
1. Start Installer
2. Click "Sources"
3. Click "Edit"
4. Click "Add"
5. Enter "http://library.pollock.ca/gutenberg_scifi"
6. Click "OK"
7. Click "Done"
8. Click "Refresh"
5. Select "Install" at the bottom of the screen. You will see "Gutenberg SciFi" is now available as an application category!
6. Before you can install a book, you will need to install GutenMark, it is under Utilities. I am using it to format the files on your phone to save me bandwidth.

There are currently 163 books in the library, in English and French. The list of books I used is from Gutenberg's SciFi CD collection. Since I can convert any Gutenberg book at this point, if there is a favourite you are looking for, please let me know.

The installation software has a preference for the HTML version of the book if it exists, however it will use the TXT version and pass it through GutenMark if it doesn't. Both versions will be split into chapters using the very handy GutenSplit.

VoIP and 911

I see that another infant has died because a VoIP call wasn't properly routed to a 911 operator. Situations like this caused some very extreme results for US VoIP carriers - Vonage wasn't allowed to accept new customers for a couple of months until they sorted it out!

Every time I see one of these stories, I try to come up with a solution. Here's what I would do for this problem.

There are a couple of problems in the current solution that need to be solved. First, the call was routed based on the customer's billing data. That isn't accurate in the VoIP world. It's not even accurate in the fixed line world! Many people have phones that are billed to PO boxes or businesses that are not the physical location of the line. Second, E911 has the ability to locate an individual caller. Without that, if the call is disconnected, or the caller cannot speak, the 911 operator has no way of determining where the person needing help is.

First, the easy one. Routing based on billing data. This is pretty easy, DON'T DO IT!

We need to find some reasonably accurate geolocation data in a VoIP call. Thankfully, it's right there. The source IP address. Home VoIP will be at the end of either a cable or DSL modem. Either is a physical port. With a static IP address, it's well defined where they are. Even with a dynamic address, the address range is still geographically limited. Definitely accurate enough to select a 911 call center.

We can make use of commercial IP address geolocation databases to route to the nearest 911 operator. There are existing commercial databases that provide just this information. However, the data may be inaccurate. So, you add the data to the call. You now have a call with location data of the call and the billing data. If they disagree, you route based on the IP address and inform the call center to verify the region information.

Perhaps it's too simple a solution. VoIP telephony people (and most Internet people) view the Internet as a cloud, they don't trust IP addresses to identify locations (Telephony people have the opposite problem, they put too much trust in the phone number.)

However, society has started using IP addresses as not only identification of location, but of a specific user too. For an example, have a look at the RIAA lawsuits in the US. The IP address is sufficient to sue an individual for copyright infringement. It is used to identify not only the location the connection is made from, but to make a reasonable assertion to the identity of the individual!

So, current commercial IP address geolocation products have the information we need to properly route 911 calls to the local call center.

How do we get E911 levels of location accuracy? To do that, we need the help of the ISPs.

Remember, everyone is connecting through DSL or cable, or some other fixed-line connection. We will deal with mobile later. That means that the ISP has a mapping between the IP address and the physical port (at least for DSL, cable too probably). However, gaining access to this data will require government regulation and probably some money changing hands (carriers bill for 911 services).

If this database is also made available to the 911 operator, then we have a complete system, a database that maps an IP address to a call center, and then on from the IP address to the physical location. There is probably even a standard that we can use, since E911 services for mobile phones have the same problems.

So, what about mobile? Well, in the mobile world, we still have the IP address to location mapping, and the carrier still has the IP address to physical location mapping. However, it is a little more indirect. There is a mapping from IP address to phone number, and then the carrier can use their existing E911 services to locate the actual device.

Is any of this even expensive code? No, it isn't. That's the interesting bit. It's all information that is added on to the existing 911 call. Even without a correct geolocation of the IP address, the call center operator can still correct the problem by asking "Where are you?". This system will be there to provide additional information to the 911 operator, not replace them. Since the 911 operator themselves are a key component of this solution, additional training will undoubtedly be needed, including a change in some of the scripts they read from. Adding the city to the address for confirmation would solve many problems.

Where does this break down?

1. VPN - If the phone is in a branch office, that routes IP traffic through a VPN to a different POP, it won't work. However, they've already got this exact same problem with corporate PBXs anyways, this isn't new.
2. Calls from outside of the jurisdiction. If someone takes the device overseas to use it for cheap (free) long distance.
3. Carriers making frequent changes to their routing rules. If a subnet moves from one region to another, 911 calls may have the wrong location until the database is updated.

Baen Free Library iPhone eBook Repository Updated!

Since STE has released v 1.4 of the iPhone EBook reader, I have finally had to update the repository to support 1.4. The major change? It now places books in ~/Media/EBooks instead of /var/root/Media/EBooks. This is because Books.app only looks in ~/ now!

I took the opportunity and added some additional features:

1. Cover Art! The books now display their cover art in the main title list.
2. Chapter Sorting! The Books.app team removed a hack they had specifically put in for Baen, it was causing performance problems. I have renamed the html files to avoid the problem.

Well, I thought it was more features than that when I started.

If you already have the book installed, it should upgrade through Installer.app. I have tested a couple of books, so I'm reasonably confident it will work.

Also, since webscriptions has a habit of updating the .zip files every week or so, I now scan their site every hour to ensure that the source plist is up to date.

Again, if you run into any problems, file a bug report over at the google code project

Truth in Advertising

If you say "Usually ships in 48 hours", it means that I should expect to see it ship in two business days. It also means that if I leave five business days, I should have the item in my little hands by the time I need it.

I'll understand if you are going to miss that deadline. Let me know, and give me the option of canceling my order. Leaving me guessing about the status of my order is frustrating and ultimately (as we'll see in a second) bad for business.

I won't be shopping at Gameplanet Store NZ again. I have placed two orders with them, the first one took an extraordinary amount of time to arrive, missing the party that I wanted it for.

I ordered a second game (mostly due to a $10 discount from the first order), and again it didn't ship inside of the 48 hour window. So, you have to ask, "What does 'Usually' mean to them?" To me, it means at least 95% of the time. How ever, two failures in two orders means it sure isn't 95% to them.

GPStore - Avoid, they're the "flying pig" of NZ software retailers.

Announcing the iPhone eBook Repository

Installing eBooks onto the iPhone for reading was nearly impossible. You needed to download third party tools, use scp, rename files, the works. To help, I decided to package existing eBooks for viewing on the iPhone.

Since this is an AppTapp repository, the books can be installed onto the iPhone from any location that the device has a valid network connection, GPRS, EDGE or WiFi.

The current repository provides installation instructions for 117 SciFi novels from the Baen Free Library, with plans to add additional sources. I am currently working on Project Gutenberg.

To make use of the repository:

1. Obtain an iPhone.
2. Jailbreak the iPhone, I recommend ziphone for this task.
3. Install Books, the iPhone ebook reader using the Installer application (http://code.google.com/p/iphoneebooks/ (1.37 only!))
4. Add the repository to Installer:
   1. Start Installer
   2. Click "Sources"
   3. Click "Edit"
   4. Click "Add"
   5. Enter "http://library.pollock.ca/baen_books"
   6. Click "OK"
   7. Click "Done"
   8. Click "Refresh"
5. Select "Install" at the bottom of the screen. You will see "Baen Free Library" is now available as an application category!

Just to clarify a couple of things. I am not providing copies of the .zip archives, they are provided to the iPhone by Webscription and Baen. The content of the files themselves are unchanged, although I have had to rename the html files to better fit in with the reader (I changed the SKU in the filename to Chapter).

To make it easy to spread these repositories around, I have uploaded the scripts I used to generate the repository to google code (iphoneebookrepo). If you would like to help, either with code, by reporting faults (thanks Ross!), or just general comments, I would appreciate it!

Take that Kindle!

AppTapp Exec bug

I released my project to a couple of friendly users. After giving it a try, one of the testers came back and told me the installer was leaving crufty directories lying around in the root directory.

I eventually tracked it down to this:

<array>
   <string>Exec</string>
   <string>/bin/mkdir -p "/var/root/Media/EBooks/Beyond World's End"</string>
</array>

Now, if this was in a shell, or passed to the OS using "system" or a similar call, it would work. However, it seems that they are using direct calls to exec. I can't really tell, because AppTapp is oddly closed source.

The directories it was creating were:

• "/var/root/Media/EBooks/Beyond
• /World's
• /End"

Fun!

Next up, I tried escaping the spaces, again with no luck. It looks like the code is splitting the parameters out through the use of spaces and ignores any attempt at grouping the packager might make.

Luckily CopyPath does the right thing, properly handling the spaces. It even creates parent directories, which is why I was using mkdir in the first place.

Now to figure out how to host it on Amazon S3. I really don't want a 2.5meg file being slurped across my cable modem every day!

Debugging an iPhone repository

The AppTapp Installer doesn't provide any feedback during a failed install or source update. This makes it very difficult to create packages. At first, it really feels like you are stumbling around in the dark.

Here is what I learned from my project.

Installer.app ignores bad XML documents. If you have one good version of your repository, and then make a change that results in a "bad" one, you will still see the good copy that has been cached on the phone. This confused me for a long time.

Always test the document for correctness first. Firefox is a quick, easy test tool. Point it at the URL, and if it displays the XML in tree format, you at least know that your XML is balanced, and that your web server is configured correctly.

Next, run Installer.app manually on the iPhone. This should let you know what is going wrong with the package. (original source)

1. make sure you are not running Installer.app
2. ssh into your iPhone (root password is "alpine")
3. for 1.1.3 firmware and higher: su - mobile
4. maximize your ssh window
5. cd /Applications/Installer.app
6. ./Installer
7. you will see Installer.app pop up on the iPhone's screen
8. reproduce the error
9. observe the debugging output from Installer.app in the ssh window

Finally, never, ever change the root password on your iPhone. You may think that because you are running an SSH server that is open to the world that it would be a good idea to change it. After all, there is a "passwd" command sitting right there in the terminal window. RESIST!

There is a problem with the 1.1.3 and later firmwares, and the version of passwd shipped with the BSD subsystem results in Springboard (the desktop shell) crashing continuously. I made this mistake and had to completely wipe the phone and restore it.

For security, install the services application, and disable the SSH server between uses. This will also help with the battery life of the phone!

Creating an iPhone AppTapp Repository.

I had a project where I needed to create a repository for AppTapp (Installer.app), the application used for over the air software installation on jailbroken iphones.

It's near impossible to find instructions on how to do this. If you google for them, all you get are "how to jailbreak your phone" and "how do I install applications" questions. Nothing about how to actually create a repository!

Thankfully, I eventually found the a wiki on ipodtouchfans with a detailed description.

The repository file is an XML document using Apple's plist format. Since it's XML it is both easy to pick up and nasty to use at the same time. The header portion of the plist file is self-explanatory, check out the wiki link. Here's the header from my test repository.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>info</key>
 <dict>
  <key>name</key>
  <string>Testing</string>
  <key>maintainer</key>
  <string>Jason Pollock</string>
  <key>contact</key>
  <string>jason@pollock.ca</string>
  <key>url</key>
  <string>http://www.pollock.ca/repos/baen</string>
  <key>category</key>
  <string>TestRepositories</string>
 </dict>
 <key>packages</key>
 <array>
        </array>
</dict>
</plist>

The individual package instructions go in the final array. However, this is a fully specified (if empty) repository. If you place it on a publicly accessible URL, people will be able to add it to Installer.app and track updates!

Next are the individual packages and installation scripts. The package themselves are separated into scripts and .zip archives. There is no provision to deliver software in any other format.

The install/uninstall scripts are part of the repository XML, not the .zip archive. This means that you can host the .zip files on a cheap, low cost volume server, while the xml document (since it is generally smaller) can be hosted closer to you. Web servers are more able to cache the repository document. Hopefully, with a single http request, the application is able to tell from the header response if the repository has changed, keeping transfers to a minimum. To keep from having someone change the .zip file on you, both a size and an md5 hash are provided in the plist file. Finally, the separation between script and installed files allows other people to provide fixed installers for your application!

The install/uninstall scripts are also XML, complete with if/then/else syntax. This is near impossible to either understand or get right due to all of the tag nesting, and chaining of siblings without explicit keyword enclosures. For example, let's look at an extremely simple "if" block to check the firmware version.

First, the logic in C, just to let you see what it should be doing:

   if (! FirmwareVersionIs("1.1.2")) {
       AbortOperation("Firmware 1.1.2 is required to update to 1.1.3.");
   }

Now for the insanely verbose XML code.

           <array>
             <string>IfNot</string>
             <array>
               <array>
                 <string>FirmwareVersionIs</string>
                 <array>
                   <string>1.1.2</string>
                 </array>
               </array>
             </array>
             <array>
               <array>
                 <string>AbortOperation</string>
                 <string>Firmware 1.1.2 is required to update to 1.1.3.</string>
               </array>
             </array>
           </array>

Completely impenetrable.

Installer.app works best when the .zip is already in the destination layout. Then the installation process is relatively simple. However, if not, each file must be individually copied into its target location, using the <CopyPath> command.

<array>
    <string>CopyPath</string>
    <string>iZoo.app/</string>
    <string>/Applications/iZoo.app</string>
</array>

There are several gotchas.

First, if you create directories, Installer.app creates them with no permissions what so ever, you will need to perform a chmod!

Second, the location of the Media tree was moved between 1.1.2 and 1.1.3. In current releases of the iPhone firmware, applications run as the user mobile. This means that the Media tree (where the music/etc is stored), is now in /var/mobile/Media instead of /var/root/Media. Since applications are installed on both 1.1.2 and later versions of the firmware, they need to work on both. To help with this, Installer.app has added support for "~", to refer to the home directory for the user. Again, however, this depends on a specific version of Installer.app being present. Personally, I chose to ignore all of this, since the application I was using didn't know about /var/mobile/Media anyways. Still, it is something that developers will need to figure out.

Third the syntax in the scripts is nasty. If you find yourself writing a fair bit of it, write a tool to do it for you. I'm sure it is possible to come up with a more human readable syntax that will generate the XML for you.

Finally, debugging your packages is painful and slow. However, that's for the next post.

Other than that, there's nothing too difficult about the whole thing. If the zip file you are trying to install is already in the target layout, it's dead simple.

Enjoy!

No, you are not a valued "Contact"

A curious thing happened to me today. I got a connection request on LinkedIn. Obviously it isn't the request itself that was interesting. It was the combination of the request and who it came from.

I haven't had a single positive experience with this individual. I have found them to be extremely rude and unprofessional.

So, why would they ask to connect to my network? Obviously they've just joined LinkedIn and have selected everyone that listed the same employer.

Tip: When you join a social networking site, don't do that.

However, it leaves me with a dilemma. This individual is pretty senior in the company.

Do I click "Accept" and hope to gain more value from their connections than I lose by having their stench rub off on me?

Do I click "Reject" and let them know that what I really think of them?

I think I'll go with the third option, and click "Archive", stashing them with all the other people I don't like. Someone else in the company that I am connected to will click "Accept", so I'll still have access to their connections, and their stink will be at least one level removed.

Ah, social networking.

iCal, how can you do this to me?

After talking to Bruce, it seems that he has some of the same concerns for timezones that I do.

In a post, he pointed to something called VTIMEZONE. So, I went looking to see if perhaps there was another angle to this whole timezone storage mess.

VTIMEZONE is a data element attached to local times in iCal. It seems that when a recurring event is passed in iCal, it MUST have a VTIMEZONE component. So far so good. However, that VTIMEZONE record MUST specify information for DST! Bad iCal!

Of course, since this information is instantly incorrect, we end up with the problem we have had before. Even worse, because the information is fully specified in the message, developers are probably correct in assuming that they can't ignore the data and substitute their own information!

Personally, I feel this is a bug in iCal. I can see why it exists, to hand-wave around the timezone differences between Windows and OSX/Unix. At least they make a reference to the Olson (on which zoneinfo is based) database.

Developers, when implementing an iCal parser, use the VTIMEZONE value you receive as a way to determine the timezone, not as an actual descriptor. Based on the information in the VTIMEZONE component, find the timezone in your own TZ database and use that name when referring to the event. Never, ever, cache the definition.

Timezones and the FUTURE!

After reading Novell's report of the DST problem in GroupWise, I understand that a lot of these are software faults, and not problems with the OS or the timezone settings themselves.

When dealing with multiple timezones, what will a developer do? They’ll convert it to UTC for storage and comparison and then convert it back for display. This solution appears to completely remove the timezone from the problem and gives the developer a nice, sortable value to store.

Everything works fine until someone changes the start or end of DST. Immediately, all of the timestamps that were previously created for that period between the old and new start (and end) dates will be out by an hour.

If you’ve got a monthly/weekly recurring meeting it will be out by an hour into perpetuity for that 1 week every year.

The real fix is to store dates/times in the originator’s timezone and then convert them on the fly. It makes sorting a pain in the ass (time is no longer a single int!), but we’ve got CPU to burn so why not?

One final gotcha with that. You CANNOT store the originator's timezone as an offset! It needs to be a name used to retrieve the TZ information. If it is stored as an offset (I'm looking at you Oracle), you have the exact same problem. Using a name means that it becomes difficult to create an index across timestamps with mixed timezones, but the values will be correct!

The lesson to take away? You cannot convert a timestamp to UTC for storage if that timestamp is in the future.

The only thing worse than being ignored...

Is having your ideas used and not being given credit.

Credit is important in the knowledge economy. Personal credibility is how you get and retain employment. Without credibility, don't expect to move up the chain.

One of the sales people I work with came up with a cool idea, it would instantly double or triple the number of customers for the company. So, he did what he thought was the smart thing. When the CEO took them out for dinner, he told him about it. He sent an email around the marketing team. He had design discussions looking for angles on his own time with interested members of the engineering team.

The marketing team went quiet. There was no indication that the company would proceed with the idea. Definitely nothing along the lines of a "thank you". Of course, based on my experience, that's to be expected.

Imagine his surprise when I went up to him this morning and said, "Congratulations on that thing! I see from the press release that you got some traction! Great!"

The hurt was visible. At first he was angry, "That jerk stole my idea", then he was incredibly hurt.

I can tell you for sure, he's another person who has had his creativity killed. Do you really think that after this experience he's going to bring his A-game to work? I've never seen the fire go out of someone before. It's a sad thing to see.

He's an insanely motivated and energetic guy. If I worked somewhere else, I would be inviting him out to lunch right now.