PalmIIIxe – Part 4 – To do lists and Memos

However, mundane the PalmPilot is in today’s modern landscape, its simplicity still intrigues me. So as I continue to try to document the various steps I have taken to bring back the technology to life, at least within my own sphere of use, the hope is that we can all get back to a point where technology really is an enabler rather than just a distraction. So here is how I handled the final two basic components offered in a Palm. To Dos and Memos.

See Part 1, Part 2, Part 3

To Do Lists

I use Things 3.0 from Cultured Code to help sort my list of item that I need to complete. As the number of To Do lists out there are large and varied, I can’t say for sure if this method would work for anyone else unless you are also using something similar, or have a workflow similar to mine. The key idea is the same as the experience gathered from working on iCal and Contacts… To get the database exported to some common format so that I can work on it with the PERL Palm modules.

The key ingredient is this project available on Github called things.sh. This very capable command is able to get all your cloud connected Things information into a CSV file which you can then parse. There isn’t really a lot of fancy magic here. However, there are a few text manipulation issues we have to deal with.

  • Long names and spaces
  • Projects and Categories

These items above are just so common in modern lists that didn’t use to exist. I had to find a way to be able to map all these fancy new capabilities into a simple category and checkbox. The decision that I made was this:- (1) All Categories and Projects will be “flattened” into a category. (2) All Project titles will have a prefix of “[P]”. (3) All incompatible items from Things will get mapped into a notes field.

(1) and (2) allows me to keep things simple, and (3) is simply how I handled iCal and Contacts. Keep in mind that Text::Unidecode and Encoding::FixLatin continues to provide the capability to help in the Chinese/English situation.

As in the To Do lists being generated from the CSV from things.sh, this was essentially a lot of file manipulation and string parsing. Nothing particularly interesting in the way the code was written in order to handle any oddities.

Memos

This was a little more complex than handling a Things CSV file. After some research, I have found that Memo syncing tends to be handled by individual text files. This works great for me as I’m all for text files. However, I’m an Evernote user and that’s definitely able to handle more than just text. So, a process has to be put in place for me to be able to get “Stuff” on my palm.

  1. I have to decide what I want in the Palm
  2. This has to be placed into a separate category in my Evernote. So a separate notebook is used e.g an Evernote Notebook called “ForPalm”
  3. I have to be able to get each individual note into a text file
  4. Convert the text file into a MemoDB file for Palm
  5. Sync that

I have to admit that the above has proven itself, over time, that it isn’t the best way to get memos into the unit, but it was for now the easiest. To be able to export the Evernote Notebook into a *.enex file requires a non-AppStore version of Evernote. Once the dump is obtained, I put it through a script I found called enex-dump. This small little tool in PHP helps to convert this dump into a folder of plain text documents. This folder is then parsed for content and transformed into the MemoDB file.

Buttons Functionality Complete

With this, the functionality of the basic 4 buttons of my palm is complete. Over time, small little fixes for display are found and fixed, but otherwise, everything is pretty stable. As long as I could export the data into the format required, be it ICS, VCard, CSV or text files, I have the capability of getting that information into my PalmIIIxe and show it.

All buttons mapped

Additional Notes

As the use of my Palm now works in tandem with my iPhone, i have noticed that To Dos and Memos tend to have the highest capacity for change. While i can hold off not syncing calendar items and contacts, To Dos and Memos then to require more immediate attention. And while the current mechanisms are 1-way sync, I’m starting to feel that there might be a need to change how I handle both these items such that 2-way sync can be achieved. I’m still weighing my options, it might require a new method, software architecture or perhaps some new way of addressing it. As with always, its a function of time I can spare for this project.

See Part 1, Part 2, Part 3

PalmIIIxe – Part 3 – Getting Contacts data into the Palm

If you have seen Part 1 and Part 2 of me trying to make use of the Palm IIIxe, this particular post would be no different. In fact, it might just be more of the same.

Contacts in Mac is not much different from iCal. We can essentially export all entries from the Contacts application into a vCard. This resulting *.vcf file can then be parsed and written into the Addressbook PDB format using PERL.

So whats different?

The key difference between Datebk and Address is (1) the availability of “categories” in Addresses, and (2) the higher likelihood of using Unicode characters. Coming from a multi-lingual society where most text are not ASCII, this was suddenly an interesting problem when parsing information in PERL and “depositing” the data into a device that is essentially ASCII only.

So… Categories

Handling categories was relatively simple. We just had to ensure that we were able to obtain the category information from the vCards and match/write them out into the same category IDs as in the PDB. If we can’t find a category entry in the vCard, then we just slotted it into “Unfiled”. The PERL libraries used were Text::vCard::Addressbook, and writing out to the address book PDB format using Palm::Address.

One other thing to note is that there are lots of items in the modern vCard format that didn’t used to be available in the Palm. The simple way to deal with all these was to just have ALL the information parsed and stored into the Notes field in the Palm. That way, I effectively don’t lose any information when looking up data, but also am able to just sort and keep the most important information visible.

Imported vCards in AddressBook and Categories Kept
Notes field keeping all vCard information intact

As with iCal, this is really a 1-way sync. Changes in the Palm wouldn’t be updated back into Contacts, but IMHO, that isn’t really required.

What about Unicode and UTF8?

This was the part that was interesting. Initially, parsing a small subset of my vCard export , handling categories and notes was successful. But when I finally exported the full data, the Palm crashed. Actually, I can’t really remember if it crashed or not, but I do remember that it didn’t go well. (*I’m writing this about 3/4 of a year I wrote the code). When I finally discovered what was wrong, it was clearly the fact that as awesome as the Palm Pilots were, it didn’t handle UTF8 too well. I had two solutions. (1) Install CJKOS as a way to get the right fonts on my device, and then get the data into the PDB using the right fonts, or (2) erase, negate the UTF8 data so that I would leave behind only the ASCII text.

Both weren’t particularly good solutions as (1) would result in me trying to ensure that the character sets are in sync all the time. Additionally, I couldn’t find CJKOS anymore. (2) was worse. I would lose data. This rendered some of the card information useless, as there are folks whom I only and have information in their native language. I have also gotten used to seeing their names in mandarin. Romanizing their names would therefore not be a good solution.

As I dug around for a solution, I finally came across a PERL module called Text::Unidecode. This seems to be an awesome little module that is able to, in the best of its ability, translate, phonetically, the words in Unicode, into its English, ASCII equivalent. And I’ll say, the results are pretty amazing. Testing on mandarin, it was able to get the Han Yu Pin Yin pronunciations of words I threw at it. Below is one of the examples.

Text::Unidecode handling the conversion of Unicode into Han Yu Pin Yin

I was impressed.

I immediately got this integrated into every field that I had across iCal and Contacts, and potentially saved me a lot of grief porting data across a modern system and a more vintage technology like the Palm.. which doesn’t understand Unicode.

With two functions down, I only have two more to go. Things and Evernote.

Update 14/1/2019: It seems that I have just came across a small bug that caused Unicode characters to be recognized as gibberish. This seems to be due to some kind of bad encoding on the text string, which my Mac is able to decode, but Text::Unidecode is unable to handle. In order to fix this, all strings are now passed through Encoding::FixLatin in order to ensure that the text is encoding correctly. Once done, Text::Unidecode performs as expected.

See Part 1, Part 2

PalmIIIxe – Part 2 – Getting iCal data into the Palm

Getting data into the Palm IIIxe is a challenge. First of all, it doesn’t have Wifi. Secondly, it uses its own data format *.PDB to store data. Which means not only do I have to find a way to sync the data over the serial port, I also have to be able to convert the data that I currently have into the compatible format.

As mentioned before in the previous post, pilot-xfer is my friend. It can help me sync data in a few ways into the Palm. Serial or network. Thank goodness that Palm at least had the sense to include Network syncing, but the pilot-link suite of tools is the one that made it possible to redirect a serial port to a network-ed system running a HotSync capable machine.

Diagram of how my Palm syncs data

Once I had this setup, getting files in and out of my Palm was possible. Not easy, but definitely possible. As shared before, the productivity utilities that I use daily are as follows.

  • iCal -> Datebk
  • Contacts -> Addressbook
  • ToDo -> Things3.0
  • Memo -> Evernote

This means that somehow, I have to convert the data from these systems into the respective PDB files that can be understood by the Palm built-in apps. I started off with iCal and Contacts as these have standard formats that we can convert into and thus transform into records that can then be written out into PDB. I’m familiar with PERL, so I started hunting for PERL packages that would handle Palm PDB files. Exporting from iCal and Contacts on Mac is easy, so there isn’t much to worry about there.

Handling ICS to Datebk PDB conversion

The PERL package I used falls under the PALM package and the ICS file parse I used is iCal::Parser. A few key things to note. What PALM can support is definitely simpler compared to what iCal can handle. So it is very important to NOT lose any data during the transformation. Therefore, all un-mappable information would go into the NOTES field in the PalmPilot. Its also important to know that the iCal Parser that I am using handles all the conversion of ICS entries into parsable calendar objects. This makes it much easier for me to be able to not worry about things like repeated events etc and just focus on the mapping. This isn’t a coding tutorial, so I wouldn’t be providing code to show how I got it done. But ultimately, I made some decisions about how things should show up on Datebk, manipulated the strings (PERL is good for things like that), provide by the parser, and wrote it out into a PDB.

High-level prices flow of the data conversion

Needless to say, it took a while for me to settle on exactly how I would like the data to show up and the abbreviations I would use, but the end result, I think, is pretty concise and usable. The screen, being only 160×160 also mean I had to choose carefully what I wanted to show. It really drives home the need to only show exactly what is required and important, and how far technology has come.

Datebk showing my iCal data
Notes field showing all other information meaningful to the event so that nothing is lost

Once the PDB file has been constructed, the next thing was to be able to move the file seamlessly to the networked HotSync machine for syncing. Making use of Dropbox to do this was no a no brainer. I previously had some Linux code that I wrote to be able to sync files, so I effectively just re-used those. I could have just used rclone to do the same. The flow below shows how the PDB file would move from the moment it has been constructed. As the files are very small, the cloud sync only really takes a moment.

Moving the file so that we can HotSync it over the network

Once the file is on the HotSync system, pilot-xfer and pilot-nredir does the rest.

Calendar Week view on iCal
View of the same week on Palm Datebk

Tada! 🙂
I now have my iCal available on my Palm.

Final bits on the conversion

Putting on the finish touches on this conversion process just requires me to now be able to automatically export the iCal entries using Apple Script, for all my calendars into a file, so that parsing script can then take over the conversion. It was at this point that I ran into some slight issues. I realised that I didn’t want to have my Mac handling the ICS file export and then running the conversion. This is largely because I didn’t want to have to re-install all those linux tools every time I change or upgrade my Mac. I wanted the tools to be on an OS whose tools are generally available as part of the system. I also wanted to ensure that all the other tools I ended up creating could follow the same structure. So I tweaked the process just a little bit….

Final ics file to Datebk syncing workflow

At the end of this, I could export any parsable file into a target folder in DropBox, and it will be MD5 checked for any changes. Once a change is detected, it would then automatically (re)create an updated PDB file which would be placed in a HotSync folder. Connecting my Palm to the RaspberryPi, which has pilot-nredir always running in a while loop, would then automatically sync, install the latest PDB file into it.

Issues?

Not much… everything I have seems to translate pretty well into text. The one thing that I have to remember that this sync is ONE-WAY. It gets iCal stuff into the Palm, but not any changes in Palm back into iCal. This means that I can’t create, edit, delete entries on the Palm and expect it to carry across to my iCal.

HOWEVER, I find that short-coming ok. I use my iPhone to keep track of my calendars anyway. The Palm is really just a way for me to not have to whip out my phone just to check a calendar entry if I’m playing games on it.

At some point in the future, I might have to re-design the entire process if I want to be able to do a 2-way sync, but it might not be worth the trouble.

See Part 1

PalmIIIxe – Part 1 – Setting up the Vintage Hardware so that it is useful again

I miss the simplicity of the Palm Pilot. Maybe I’m getting old, but there is something about it that I can’t help but to feel that its exactly what I need. No distractions of a modern GUI, and just straight out text. However, I no longer have a unit anymore. In progressing from the various PDAs, handhelds and mobile phones, I have either sold off all my units or had them break down on me. As such, I set out to select and purchase a Palm series PDA in 2018.

The selection criteria as follows. It had to be (1) easy to maintain, (2) easy to fix, (3) fun to use. This meant that no has Li-ion battery, with primitive parts to replace or change. Straight forward to rip apart and replace parts and to hack too! I ultimately settled on the Palm IIIxe. Its AAA batteries, push buttons and a screw on back meant that it met all my above criterias. (You can probably find the Palm IIIxe specs somewhere on the internet). The vintage set cost about USD$50 bucks. And just a fun project, I bought it with the intention to make it useful again, one way or another, and make it fit with the modern world that we live in.

PalmIIIxe Box
What the PalmIIIxe provided

With the Palm in hand after getting it from Ebay, I bought some sync cables (it was really hard to find) with the intention to hack them in the future, I started to get the unit ready for use. A few things presented itself as a problem immediately. (1) I can’t find a serial port, (2) I don’t have windows (for HotSync), and (3) I needed software.

Palm IIIxe PDA

(1) and (2) was challenging as I chanced upon the latest Palm OS 4.0 which the PalmIIIxe was compatible with, and needed the serial port to perform the firmware upgrade. I finally managed to have a friend dig up an old IBM Thinkpad with serial port available and flashed it. Phew!

Palm OS3 boot screen
Palm OS4 boot screen

Software was yet another challenge. As I am not primarily a windows user, (I’m a linux fan but use Mac for work), most of the default software don’t work. So Pilot-Xfer was my friend. Digging up an older DB9 Serial (RS232) to USB convertor, and using Pilot-Xfer in network mode, I managed to sync my device to a Linux VM I created just for the palm. So that solved the basic software syncing issue. And now, i started to hunt for the games that I used to enjoy.

Snapshot of the pilot-xfer MAN page

That was tough….

Almost all games and utilities where either obsolete, or the stores gone. PalmGear no longer exists, and its impossible to register some of the shareware games anymore. Some utilities that I used to use are now impossible to find. But I still managed to find a trove of stuff kept by some as an archive. Those I downloaded in entirety, so that I can hold them only own storage, and hopefully, overtime, sort through them… but more on that another time.

In the short term, I installed a bunch of free software such as the following for the Palm to be a gaming machine and some basic utilities as a start.

  • LauncherIII
  • Gobble
  • Free-Tris
  • BigTwo (my Fav!)
  • EasyCalc
  • Keyring
  • Metro
  • PalmXQ

But one thing that was still missing is the data associated with the built-in apps. DateBook, AddressBook, ToDo, Memo. In the modern day and age where everything is in the cloud, the one biggest thing I missed on the Palm apps is its simplicity and single-mindedness… something I admire and appreciated. However, in keeping with the times, all my events are kept on a Mac. My ToDo items are stored in Things (by Cultured Code). Notes in Evernote. So the question to essentially answer was…

“How do I get my Calendar, ToDos, Contacts and Memos from my modern desktop using iCal, Contacts on Mac, Things from Cultured Code and Evernote, into my Palm?”

There are no Conduits to do such things, and obviously, the pilot-suite on linux are so old that they don’t support such new tools. So the only answer was… roll your own code.

Till next time…

PS: I’m pretty sure no one is going to fry me over blogging about this as its just a hobby and it doesn’t really have any conflict of interest on my job anyway. Palm is pretty much defunct on its PDA product line anyway. So I’m going to just write about this and hopefully don’t get into any trouble.