Tuesday, April 29, 2008

Baen Free Library iPhone eBook Repository Updated!

Since STE has released v 1.4 of the iPhone EBook reader, I have finally had to update the repository to support 1.4. The major change? It now places books in ~/Media/EBooks instead of /var/root/Media/EBooks. This is because Books.app only looks in ~/ now!

I took the opportunity and added some additional features:

  1. Cover Art! The books now display their cover art in the main title list.
  2. Chapter Sorting! The Books.app team removed a hack they had specifically put in for Baen, it was causing performance problems. I have renamed the html files to avoid the problem.

Well, I thought it was more features than that when I started.

If you already have the book installed, it should upgrade through Installer.app. I have tested a couple of books, so I'm reasonably confident it will work.

Also, since webscriptions has a habit of updating the .zip files every week or so, I now scan their site every hour to ensure that the source plist is up to date.

Again, if you run into any problems, file a bug report over at the google code project

Tuesday, April 08, 2008

Truth in Advertising

If you say "Usually ships in 48 hours", it means that I should expect to see it ship in two business days. It also means that if I leave five business days, I should have the item in my little hands by the time I need it.

I'll understand if you are going to miss that deadline. Let me know, and give me the option of canceling my order. Leaving me guessing about the status of my order is frustrating and ultimately (as we'll see in a second) bad for business.

I won't be shopping at Gameplanet Store NZ again. I have placed two orders with them, the first one took an extraordinary amount of time to arrive, missing the party that I wanted it for.

I ordered a second game (mostly due to a $10 discount from the first order), and again it didn't ship inside of the 48 hour window. So, you have to ask, "What does 'Usually' mean to them?" To me, it means at least 95% of the time. How ever, two failures in two orders means it sure isn't 95% to them.

GPStore - Avoid, they're the "flying pig" of NZ software retailers.

Friday, March 28, 2008

Announcing the iPhone eBook Repository

Installing eBooks onto the iPhone for reading was nearly impossible. You needed to download third party tools, use scp, rename files, the works. To help, I decided to package existing eBooks for viewing on the iPhone.

Since this is an AppTapp repository, the books can be installed onto the iPhone from any location that the device has a valid network connection, GPRS, EDGE or WiFi.

The current repository provides installation instructions for 117 SciFi novels from the Baen Free Library, with plans to add additional sources. I am currently working on Project Gutenberg.

To make use of the repository:

  1. Obtain an iPhone.
  2. Jailbreak the iPhone, I recommend ziphone for this task.
  3. Install Books, the iPhone ebook reader using the Installer application (http://code.google.com/p/iphoneebooks/ (1.37 only!))
  4. Add the repository to Installer:
    1. Start Installer
    2. Click "Sources"
    3. Click "Edit"
    4. Click "Add"
    5. Enter "http://library.pollock.ca/baen_books"
    6. Click "OK"
    7. Click "Done"
    8. Click "Refresh"
  5. Select "Install" at the bottom of the screen. You will see "Baen Free Library" is now available as an application category!

Just to clarify a couple of things. I am not providing copies of the .zip archives, they are provided to the iPhone by Webscription and Baen. The content of the files themselves are unchanged, although I have had to rename the html files to better fit in with the reader (I changed the SKU in the filename to Chapter).

To make it easy to spread these repositories around, I have uploaded the scripts I used to generate the repository to google code (iphoneebookrepo). If you would like to help, either with code, by reporting faults (thanks Ross!), or just general comments, I would appreciate it!

Take that Kindle!

Wednesday, March 26, 2008

AppTapp Exec bug

I released my project to a couple of friendly users. After giving it a try, one of the testers came back and told me the installer was leaving crufty directories lying around in the root directory.

I eventually tracked it down to this:


<array>
   <string>Exec</string>
   <string>/bin/mkdir -p "/var/root/Media/EBooks/Beyond World's End"</string>
</array>

Now, if this was in a shell, or passed to the OS using "system" or a similar call, it would work. However, it seems that they are using direct calls to exec. I can't really tell, because AppTapp is oddly closed source.

The directories it was creating were:

  • "/var/root/Media/EBooks/Beyond
  • /World's
  • /End"

Fun!

Next up, I tried escaping the spaces, again with no luck. It looks like the code is splitting the parameters out through the use of spaces and ignores any attempt at grouping the packager might make.

Luckily CopyPath does the right thing, properly handling the spaces. It even creates parent directories, which is why I was using mkdir in the first place.

Now to figure out how to host it on Amazon S3. I really don't want a 2.5meg file being slurped across my cable modem every day!

Tuesday, March 25, 2008

Debugging an iPhone repository

The AppTapp Installer doesn't provide any feedback during a failed install or source update. This makes it very difficult to create packages. At first, it really feels like you are stumbling around in the dark.

Here is what I learned from my project.

Installer.app ignores bad XML documents. If you have one good version of your repository, and then make a change that results in a "bad" one, you will still see the good copy that has been cached on the phone. This confused me for a long time.

Always test the document for correctness first. Firefox is a quick, easy test tool. Point it at the URL, and if it displays the XML in tree format, you at least know that your XML is balanced, and that your web server is configured correctly.

Next, run Installer.app manually on the iPhone. This should let you know what is going wrong with the package. (original source)

  1. make sure you are not running Installer.app
  2. ssh into your iPhone (root password is "alpine")
  3. for 1.1.3 firmware and higher: su - mobile
  4. maximize your ssh window
  5. cd /Applications/Installer.app
  6. ./Installer
  7. you will see Installer.app pop up on the iPhone's screen
  8. reproduce the error
  9. observe the debugging output from Installer.app in the ssh window

Finally, never, ever change the root password on your iPhone. You may think that because you are running an SSH server that is open to the world that it would be a good idea to change it. After all, there is a "passwd" command sitting right there in the terminal window. RESIST!

There is a problem with the 1.1.3 and later firmwares, and the version of passwd shipped with the BSD subsystem results in Springboard (the desktop shell) crashing continuously. I made this mistake and had to completely wipe the phone and restore it.

For security, install the services application, and disable the SSH server between uses. This will also help with the battery life of the phone!

Creating an iPhone AppTapp Repository.

I had a project where I needed to create a repository for AppTapp (Installer.app), the application used for over the air software installation on jailbroken iphones.

It's near impossible to find instructions on how to do this. If you google for them, all you get are "how to jailbreak your phone" and "how do I install applications" questions. Nothing about how to actually create a repository!

Thankfully, I eventually found the a wiki on ipodtouchfans with a detailed description.

The repository file is an XML document using Apple's plist format. Since it's XML it is both easy to pick up and nasty to use at the same time. The header portion of the plist file is self-explanatory, check out the wiki link. Here's the header from my test repository.


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>info</key>
 <dict>
  <key>name</key>
  <string>Testing</string>
  <key>maintainer</key>
  <string>Jason Pollock</string>
  <key>contact</key>
  <string>jason@pollock.ca</string>
  <key>url</key>
  <string>http://www.pollock.ca/repos/baen</string>
  <key>category</key>
  <string>TestRepositories</string>
 </dict>
 <key>packages</key>
 <array>
        </array>
</dict>
</plist>

The individual package instructions go in the final array. However, this is a fully specified (if empty) repository. If you place it on a publicly accessible URL, people will be able to add it to Installer.app and track updates!

Next are the individual packages and installation scripts. The package themselves are separated into scripts and .zip archives. There is no provision to deliver software in any other format.

The install/uninstall scripts are part of the repository XML, not the .zip archive. This means that you can host the .zip files on a cheap, low cost volume server, while the xml document (since it is generally smaller) can be hosted closer to you. Web servers are more able to cache the repository document. Hopefully, with a single http request, the application is able to tell from the header response if the repository has changed, keeping transfers to a minimum. To keep from having someone change the .zip file on you, both a size and an md5 hash are provided in the plist file. Finally, the separation between script and installed files allows other people to provide fixed installers for your application!

The install/uninstall scripts are also XML, complete with if/then/else syntax. This is near impossible to either understand or get right due to all of the tag nesting, and chaining of siblings without explicit keyword enclosures. For example, let's look at an extremely simple "if" block to check the firmware version.

First, the logic in C, just to let you see what it should be doing:


   if (! FirmwareVersionIs("1.1.2")) {
       AbortOperation("Firmware 1.1.2 is required to update to 1.1.3.");
   }

Now for the insanely verbose XML code.


           <array>
             <string>IfNot</string>
             <array>
               <array>
                 <string>FirmwareVersionIs</string>
                 <array>
                   <string>1.1.2</string>
                 </array>
               </array>
             </array>
             <array>
               <array>
                 <string>AbortOperation</string>
                 <string>Firmware 1.1.2 is required to update to 1.1.3.</string>
               </array>
             </array>
           </array>

Completely impenetrable.

Installer.app works best when the .zip is already in the destination layout. Then the installation process is relatively simple. However, if not, each file must be individually copied into its target location, using the <CopyPath> command.


<array>
    <string>CopyPath</string>
    <string>iZoo.app/</string>
    <string>/Applications/iZoo.app</string>
</array>

There are several gotchas.

First, if you create directories, Installer.app creates them with no permissions what so ever, you will need to perform a chmod!

Second, the location of the Media tree was moved between 1.1.2 and 1.1.3. In current releases of the iPhone firmware, applications run as the user mobile. This means that the Media tree (where the music/etc is stored), is now in /var/mobile/Media instead of /var/root/Media. Since applications are installed on both 1.1.2 and later versions of the firmware, they need to work on both. To help with this, Installer.app has added support for "~", to refer to the home directory for the user. Again, however, this depends on a specific version of Installer.app being present. Personally, I chose to ignore all of this, since the application I was using didn't know about /var/mobile/Media anyways. Still, it is something that developers will need to figure out.

Third the syntax in the scripts is nasty. If you find yourself writing a fair bit of it, write a tool to do it for you. I'm sure it is possible to come up with a more human readable syntax that will generate the XML for you.

Finally, debugging your packages is painful and slow. However, that's for the next post.

Other than that, there's nothing too difficult about the whole thing. If the zip file you are trying to install is already in the target layout, it's dead simple.

Enjoy!

Thursday, March 20, 2008

No, you are not a valued "Contact"

A curious thing happened to me today. I got a connection request on LinkedIn. Obviously it isn't the request itself that was interesting. It was the combination of the request and who it came from.

I haven't had a single positive experience with this individual. I have found them to be extremely rude and unprofessional.

So, why would they ask to connect to my network? Obviously they've just joined LinkedIn and have selected everyone that listed the same employer.

Tip: When you join a social networking site, don't do that.

However, it leaves me with a dilemma. This individual is pretty senior in the company.

Do I click "Accept" and hope to gain more value from their connections than I lose by having their stench rub off on me?

Do I click "Reject" and let them know that what I really think of them?

I think I'll go with the third option, and click "Archive", stashing them with all the other people I don't like. Someone else in the company that I am connected to will click "Accept", so I'll still have access to their connections, and their stink will be at least one level removed.

Ah, social networking.

Wednesday, March 19, 2008

iCal, how can you do this to me?

After talking to Bruce, it seems that he has some of the same concerns for timezones that I do.

In a post, he pointed to something called VTIMEZONE. So, I went looking to see if perhaps there was another angle to this whole timezone storage mess.

VTIMEZONE is a data element attached to local times in iCal. It seems that when a recurring event is passed in iCal, it MUST have a VTIMEZONE component. So far so good. However, that VTIMEZONE record MUST specify information for DST! Bad iCal!

Of course, since this information is instantly incorrect, we end up with the problem we have had before. Even worse, because the information is fully specified in the message, developers are probably correct in assuming that they can't ignore the data and substitute their own information!

Personally, I feel this is a bug in iCal. I can see why it exists, to hand-wave around the timezone differences between Windows and OSX/Unix. At least they make a reference to the Olson (on which zoneinfo is based) database.

Developers, when implementing an iCal parser, use the VTIMEZONE value you receive as a way to determine the timezone, not as an actual descriptor. Based on the information in the VTIMEZONE component, find the timezone in your own TZ database and use that name when referring to the event. Never, ever, cache the definition.

Tuesday, March 18, 2008

Timezones and the FUTURE!

After reading Novell's report of the DST problem in GroupWise, I understand that a lot of these are software faults, and not problems with the OS or the timezone settings themselves.

When dealing with multiple timezones, what will a developer do? They’ll convert it to UTC for storage and comparison and then convert it back for display. This solution appears to completely remove the timezone from the problem and gives the developer a nice, sortable value to store.

Everything works fine until someone changes the start or end of DST. Immediately, all of the timestamps that were previously created for that period between the old and new start (and end) dates will be out by an hour.

If you’ve got a monthly/weekly recurring meeting it will be out by an hour into perpetuity for that 1 week every year.

The real fix is to store dates/times in the originator’s timezone and then convert them on the fly. It makes sorting a pain in the ass (time is no longer a single int!), but we’ve got CPU to burn so why not?

One final gotcha with that. You CANNOT store the originator's timezone as an offset! It needs to be a name used to retrieve the TZ information. If it is stored as an offset (I'm looking at you Oracle), you have the exact same problem. Using a name means that it becomes difficult to create an index across timestamps with mixed timezones, but the values will be correct!

The lesson to take away? You cannot convert a timestamp to UTC for storage if that timestamp is in the future.

Monday, February 11, 2008

The only thing worse than being ignored...

Is having your ideas used and not being given credit.

Credit is important in the knowledge economy. Personal credibility is how you get and retain employment. Without credibility, don't expect to move up the chain.

One of the sales people I work with came up with a cool idea, it would instantly double or triple the number of customers for the company. So, he did what he thought was the smart thing. When the CEO took them out for dinner, he told him about it. He sent an email around the marketing team. He had design discussions looking for angles on his own time with interested members of the engineering team.

The marketing team went quiet. There was no indication that the company would proceed with the idea. Definitely nothing along the lines of a "thank you". Of course, based on my experience, that's to be expected.

Imagine his surprise when I went up to him this morning and said, "Congratulations on that thing! I see from the press release that you got some traction! Great!"

The hurt was visible. At first he was angry, "That jerk stole my idea", then he was incredibly hurt.

I can tell you for sure, he's another person who has had his creativity killed. Do you really think that after this experience he's going to bring his A-game to work? I've never seen the fire go out of someone before. It's a sad thing to see.

He's an insanely motivated and energetic guy. If I worked somewhere else, I would be inviting him out to lunch right now.

Tuesday, December 11, 2007

Camera Phones are Everywhere

If you doubt the ubiquity of cameraphones, check out this picture:

Girl with cameraphone at museum

You've got a little girl at a museum in Belgrade, taking a picture of a cool rock with a cameraphone. Five years ago, you would never see a phone with a camera, let alone a child with one. It will be interesting to see what happens now that we can assume that everyone has both a phone and a camera.

Cool times ahead!

Thursday, November 29, 2007

ActiveRecord Columns

It can be hard to keep up on side projects, there are just so many other things around the house demanding attention. This week, I've been spending my time demolishing a brick BBQ/planter in the back yard. I've removed 3 cubic meters of dirt and brick! Last night I finished that, so I started to think about the side project I'm working on. As I did, I thought back to the WTF moment I wrote about a while ago, and I was afraid that I was wrong. Totally wrong. Idiotically wrong in a way that would demonstrate that I am a complete total and utter Ruby on Rails noob.

It's interesting about blog posts. I've added Feedburner so I have a bit better an idea of how many people are reading this. Before, I thought it was just a couple of friends and then only when I bugged them, "Hey, did you see my new post?". Now, it seems I have a readership (albeit minimal).

That means that any mistakes I make here are really, really public and enduring. Of course, for me that's part of the point of the blog. I want to be able to go back and see how my ideas change over the years.

Anyways, back to Rails. I was afraid that this code was wrong:

def update_my_status(user, status) if (a_party == user) self.a_party_status = status else self.b_party_status = status end end

I was sitting there thinking, "Wait a second, Ruby says that data members have an '@' in front of them!", quickly followed by, "Nuts, so, do I edit or delete the previous post?"

Just to be sure, I went and fired up the rails console. Very, handy, much recommended. I modified the code to look like this:

class Call < ActiveRecord::Base def my_status(user) if (@a_party == user) @a_party_status else @b_party_status end end ...

I then called it in the console:

>> call = Call.find(:first) => # >> call.my_status(2) => nil

Whew, off the hook. It seems that Ruby has "method missing" functionality, but not "member missing" functionality. So, when I put an '@' in front of it, ActiveRecord doesn't know to look up the value in @attributes. Interesting, and a bit more learned about the internals of ActiveRecord and Rails.

Wednesday, November 14, 2007

Lying, redux.

Happened to me today:

"The NFS mounts are all broken, what's changed on the server?"
"Nothing"
"Well, obviously something's changed, or else it would work. What's changed?"
"Nothing"
...
Several more times back and forth...
"Listen, it's pretty [bleeping] obvious that you have [bleeping] changed something. What the [bleep] have you changed so that I can fix my [bleeping] systems and get some [bleeping] work done!"
"Take a chill pill, and relax dude!"

Bull? Meet Mr. Red Flag.

The actual problem? They changed the directory layout of the NFS mount, so everything moved around. Amazingly, he was surprised when everything stopped working.

Lesson? I need to realise that although the guy is lying his ass off, I need to walk away instead of rubbing his nose in it. I'm not going to get a resolution. I need to follow my own advice, and give people back their monkeys.

So, in the future, I'll just send a polite email (CC'ing my supervisor, and their supervisor), explaining why no one in the entire support organisation is able to get any work done. Hopefully they'll get it resolved by the end of the day.

Oh wait, just yesterday I gave someone advice because they had a technical manager doing exactly that! What did I say? Oh yeah, "He's looking for a reason to slip his project and he's looking to blame you. He should know how to manage his own development systems."

So, either I look like a dick because I put the beat down on some poor overworked IT guy who seems to have been here late screwing it up in the first place (and then bald-faced lying about changing anything), or I look like a dick because I'm looking to blame someone for my schedule slippages.

Hmm. I think I'll walk away and talk to someone else in the team next time. That guy has managed to flip my idiot bit.

Monday, November 12, 2007

Google's Android

I was looking at the docs for the SDK today. I'm interested to see if I need to get an OpenMoko to play with this sucker. However, I haven't been able to find the OS itself, just references to the eclipse plugin.

Even worse is this quote from the FAQ:

Android does not currently support development of third party applications in native code (C/C++).

Not as open as we had all hoped. I'll keep looking, I hope it gets better.

Sunday, November 11, 2007

Recovering a corrupted Akregator feed list

Akregator has an annoying habit of losing your configured feeds when it shuts down abnormally (as in whenever it's "killed"). It's supposed to have a backup of the file, but it manages to truncate that file too.

However, I've found a second backup of the file, ~/.kde/share/apps/akregator/Archive/feedlistbackup.mk4 It contains some binary data at the top, and then appears to have a several different versions of the opml file.

To recover:

cd ~/.kde/share/apps/akregator/Archive/
cp feedlistbackup.mk4 ../data/protect_my_backup
[shut down akregator]
cp feedlistbackup.mk4 feeds.opml
vi feeds.opml
[remove the binary data at the top of the file, leaving the XML declaration tag and the <opml>...</opml> section. Remove everything from the first </opml> to the end of the file.]
[restart akregator]

Just remember, copy the file as soon as you see the problem, if you shut akgregator down, it will overwrite the backup file when it manages to exit properly.

If that "feedlistbackup.mk4" doesn't look right, you can always try to recover the opml file from the list of sites in the archive. However, it won't be organised into folders, and it will probably have all the feeds you have ever subscribed to.

Wednesday, October 31, 2007

No I don't want to unlock your iPhone for you

Just because I've got one, and I love it, doesn't mean I'll help you with yours. It was hard enough for me to unlock mine that my recommendation on the device is this:

"If you aren't a developer familiar with SSH, firmware, networks, bootloaders, and Unix go buy something else. This device is not for you."

That includes buying one that is already unlocked. You will eventually screw up and be back to using ssh and command line shells.

I've got my own projects, and I really don't want to spend my life unlocking other people's phones when I could be having fun.

So, no, I don't want to help you unlock your phone.

The only person I will help is my wife, and that's because, well, I love her. If you aren't her, I don't love you enough, and you can't afford my hourly rate.

Monday, October 29, 2007

Meetings

I've never been in a meeting that was longer than one hour, or had more than four people that achieved anything. I'm being generous with the hour, too.

Why do long meetings continue to exist? I'm becoming more and more convinced that long meetings are a power play. They exist to re-inforce the importance of the person who was able to convince everyone else to show up. Nothing else. Who needs a four hour meeting attended by 20 people about a project that's late?

I'm curious to hear about situations where they've had a good meeting (decisions made) that lasted more than an hour or a large number of attendees. Do they work for you?

Wednesday, October 24, 2007

Rails, models and modifying columns

I had my first WTF moment with Ruby and Rails over the past couple of days. I'm working on a personal project (you'll see it soon), and I'm using Rails. It's an opportunity to learn a new language, and see if the buzz around Rails is true.

Anyways, I was having problem with a model, here's the code I had:

class Call < ActiveRecord::Base
  def my_status(user)
    if (a_party == user)
      a_party_status
    else
      b_party_status
    end
  end

  def update_my_status(user, status)
    if (a_party == user)
      a_party_status = status
    else
      b_party_status = status
    end
  end
end

My controller looked like this:

 @call = Call.find(params[:id])
 @call.update_my_status(current_user(),params[:status])

This didn't work. No matter what I did, I couldn't modify call. I put a tonne of debug lines in, and eventually got to this:


 @call = Call.find(params[:id])
 @call.update_my_status(@call.a_party, true);
 
 if !@call.a_party_status 
   @call.a_party_status = true
   if @call.a_party_status
      # Hrm, first method didn't work, but the second one did?
   else
      # They both failed!
   end
 end

Amazingly, setting the value outside of the class worked, but setting it inside of the method in the class didn't. I was stumped. Google wasn't any help, it seems this is something that no one else runs into as a problem.

Finally, I found a tutorial that showed me what was going wrong, even though it didn't say it explicitly... It had the following example code:

>> player = Player.new
=> #<Player:0x28ef720 @attributes={"team_id"=>nil, "name"=>nil, "number"=>nil,
"position"=>nil}, 
@new_record=true>

Looking at that, it all clicked. ActiveRecord makes extensive use of the "method missing" functionality available in Ruby. Essentially, it looks at what you are trying to invoke and performs the action on the fly if Ruby can't figure it out itself. That explains what is going on with the "update_my_status" method. Since it is an assignment, Ruby will create a local variable, "method missing" isn't invoked and the class attributes are never changed! The solution? Put "self." in front of the attribute assignments:

  def update_my_status(user, status)
    if (a_party == user)
      self.a_party_status = status
    else
      self.b_party_status = status
    end
  end

That was extremely frustrating. I need to see if there is a "warnings as errors" option. It's also an indication that I need to start working on my own stuff earlier in the evening. My brain seems to shut down after 9:30.

Oddly, it seems that I'm the only one who has this problem. After talking a friend, he said, "I always put self. in front of everything out of convention".

Nuts.

Oh, and Rails? It Rocks.

Tuesday, October 16, 2007

Hey, I found an iPhone bug!

This one would be fun to track down. :) After a week of use, I noticed that the volume control (and slider) weren't as responsive as they used to be. The visual control was no longer smooth, tending to move forward and back. My guess is that the animation was semi-predictive, and then corrected for the true value reported by the hardware. Since there was an increasing lag between the control and the visual update, the phone would appear to visually jerk the volume display. A quick reset and the problem's all gone. I wonder if the 1.1.1 touch has the same problem? I wonder what interupts are firing when a keypress/volume change happens?

Thursday, October 11, 2007

Testing and Testers.

Q: What do you do with a tester that feels his responsibility begins and ends with reporting a fault? Refusing to be involved in the resolution, only in the discovery and reporting.

A: Sack their sorry-ass and hire yourself someone who knows what their job really is.