PalmIIIxe Connection to TTL UART on Linux – Part 1

Serial ports are obsolete. So trying to sync the PalmIII to a modern desktop becomes a pain when you need to convert the serial port to a USB. The use of an adapter is to unwieldy to carry about, so the hunt for a PalmIII serial to USB cable begins…. and it ends rather quickly. It is very hard to find, and very expensive. So comes the consideration to make one yourself. Keep in mind that what I wanted to do was to create a cheap Palm to USB serial connector so that I could have PalmOS talk to a linux… for the purpose of… whatever.

Huge DB9 Serial to USB adapter. Expensive too.

Initial Research

On browsing the internet, there is no lack of information when it comes to documenting the PIN outs of the PalmIII serial port. I have reproduced the one that I found the most simple/useful here.

PalmIII Pins and its uses

This is just great, as it also seems to coincide with most of what other people have done with their old PalmIIIs, using it as a serial terminal. A lot of the mods out there seem to simply open up the Palm and solder stuff directly to the TX and RX lines and hooking it up to their Raspberry PI or otherwise. I didn’t want to do that as I didn’t want to “destroy” my Palm. I still wanted to be able to use it for games and to bring it around (thats what I wrote all the sync software for isn’t it?). So the goal was to have a way that my palm can interface with a modern linux system, over USB, get on the internet etc, so that I do fun stuff with it, and get continue to get the same goodness of the original Palm for use as a PDA.

This is where I made my first mistake.

Mistake 1: Not understanding Serial

I have mistakenly thought that it is simply enough to connect the TX/RX lines. But it turns out that TTL levels are very different from RS232 on a DB9 connector. The former ranges from 0-5V and the latter can swing from -9V to +9V or more. Even though both are serial ports. Hooking up directly shows gibberish on PocketTerm (by Plushworks), but it definitely shows something. This means that the first thing I had to do was to try and convert the RS232 signal to TTL. Upon digging a little more, I came across a connector for Serial to TTL. So I ordered one to try.

RS232 to TTL convertor

Problem 1: The Palm Serial Connector

In order to be able to create a direct connection from my PalmIII Hotsync Cable to TTL Serial in a small form, and do away with bulky connectors, I had to desolder the DB9, and directly interface my Palm connector to the serial port. Thats where the connection diagram proved useful. However, I didn’t want to destroy my Palm cable by cutting it. So I had to have a sustainable way to re-create the Hotsync connector so I could build things from it in the future. For that, I designed, with some interactions, an acrylic based connector which I can laser cut, then hook up at low cost. Its not fantastic, but it works.

Two self laser cut connectors for the PalmIIIxe. Connection isn’t fantastic, but it works.

There are a few issues however, that the height of the pins has to be exact. Otherwise the connection to the palm is flaky. So while doing this helped me get past certain roadblocks, using these connectors wasn’t the way I ultimately ended up with. More on that another time.

Hooking things up

Once I had the connector made, with breakout for all the lines, I followed the diagram below and started connecting it to the RS232 to TTL serial adapter. Putting everything on a breadboard, I decided to go a step further and actually put in some LEDs in order to have some visuals when there is traffic flowing or when the port is open.

PalmIII Connection to DB9
Serial to TTL convertor based on Max3232. Note that yellow TX line connects to TX on UART0
Breadboard with LEDs to show activity on Serial over TTL, UART0 connection

The sketch below shows how I hooked up only 4 lines to the TTL and had my first successful sync with pilot-xfer. The connecting system is a Cubietruck which has an onboard TTL serial (UART0) which I connected to. The resulting port on the UART is /dev/ttyS0. The testing software used was, on the Palm side PocketTerm, and on the Linux side, Minicom. The settings used to test are documented in the sketch below.

Cubietruck UART0 wiring. Note that VCC comes from another 5V pin.
Serial to TTL wiring and testing software settings

The photo below shows the hardware build and connectivity on the PalmIII side. Notice that I have a breakout board on the Palm end with some LEDs which I subsequently added to help my visually identify the state of the serial connection. So this allows me to quickly identify connection behaviour and problems across the Palm the TTL conversion. I also added some connector pins so that I can use jumper wires to hook things into the breadboard without having to re-solder all the time. Another video below shows the process of a Hotsync…. just some blinking lights there. But I was happy that this was working.

Final Palm and Cubietruck connection over TTL on UART0
Blinking LED lights during a sync over TTL

Further testing with login services on Linux

In general, to be able to offer any kind of service over the serial port, it must not be “taken up” by any other service. So I had to ensure that there are no other boot up service that is using it. As I was using a Cubietruck with the Armbian image, I just had to disable the console from the boot environment.

On Cubietruck, Armbian Distribution
delete “console” in armbianEnv.txt
or change
“console=display”

Just to bring it a step further, once I was able to establish a serial connection at 9600 or 19200 Baud to the Linux system, I tried offering login services on the Serial Port. In may posts, I have noticed that folks are trying all sorts of setup to be able to permanently offer a login prompt over serial. However, this is not something that I wish to do. I wanted to be able to (1) know exactly the service that I want to offer, and (2) have control over when I offer it. The following command provides a login prompt over the specific serial port.

On Cubietruck, Disable Serial Console on Boot
cd /etc/systemd/system
systemctl mask serial-getty@ttyS0.service
sync
reboot

Once done, you should be able to get to a login prompt when connecting over the serial line.

Logging in to a shell on the Palm

Problem 2: Maximum Baud Rate of 115200 is Unavailable

While everything works on 19200 on Minicom and PocketTerm, but when it comes to Hotsync, for some reason, I can’t get it to go above 19200 Baud rate. This holds true for Hotsync, but not the Minicom+PocketTerm combination. All research on the internet doesn’t seem to talk about why. This was a head scratcher for me. Every thing I have read seems to only say “4 lines i.e. Vcc, And, Tx, Rx, and it works”. Poking around the settings on the serial port in linux also didn’t seem to help. So while I was happy that I could now perform a Hotsync at 19200, I wasn’t completely happy as I couldn’t use the max rate of 115200.

Problem 3: Trying to Setup PPP from Palm to Linux does not work

Trying to survive on 19200 Baud rate, I tried to setup PPP on my linux system such that I would be able to get a network connection routed from my Palm to the internet via the linux machine. This was working using my bulky Serial to USB adapter, and I wanted to replicate this with my connection. However, this wasn’t the case. Trying to setup a connection continually fails, with the server side being stuck at the following message.

sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x7ba91055> <pcomp> <accomp>]

All this seems to indicate something was wrong. It seems that having just Tx and Rx lines being connected isn’t enough. I started to wonder if the RTS and CTS lines needs to be connected as well. And if DTR needed to be connected.

End of Part 1

At this stage, lets review what was accomplished/learnt/questions:-
1) I have managed to get my Palm connected to a UART TTL Serial Port at limited speed
2) A Max3232 logic convertor is required to convert RS232 signals to TTL
3) The base Palm Baud rate seems to be 19200.
4) Connecting with just TX and RX provides connectivity over serial at the base baud rate, but can’t seem to get to the max speed of 115200 for Networking or Hotsync
5) Do we need all 10 pins to be connected as is in the diagram to be able to replicate the functions of the original HotSync cable?
6) We need to have a clean, stable and extensible way of connecting the Palm Serial port to a breakout board for further development

Points 3-5 seems to be interrelated, and in order to solve it, I had to first have my serial connector be reliable as stated in point 6. As noted earlier, my self made connector seems a little flaky when trying to connect to the serial pins. There was therefore a need to change the way I connected to the serial pins before I tried to tackle this.

I think that, point (6), deserves another post. So till my next installation, where I will highlight what I did to improve the connectivity situation to the 10 pins on my Palm.

PalmIIIxe – Part 4 – To do lists and Memos

However, mundane the PalmPilot is in today’s modern landscape, its simplicity still intrigues me. So as I continue to try to document the various steps I have taken to bring back the technology to life, at least within my own sphere of use, the hope is that we can all get back to a point where technology really is an enabler rather than just a distraction. So here is how I handled the final two basic components offered in a Palm. To Dos and Memos.

See Part 1, Part 2, Part 3

To Do Lists

I use Things 3.0 from Cultured Code to help sort my list of item that I need to complete. As the number of To Do lists out there are large and varied, I can’t say for sure if this method would work for anyone else unless you are also using something similar, or have a workflow similar to mine. The key idea is the same as the experience gathered from working on iCal and Contacts… To get the database exported to some common format so that I can work on it with the PERL Palm modules.

The key ingredient is this project available on Github called things.sh. This very capable command is able to get all your cloud connected Things information into a CSV file which you can then parse. There isn’t really a lot of fancy magic here. However, there are a few text manipulation issues we have to deal with.

  • Long names and spaces
  • Projects and Categories

These items above are just so common in modern lists that didn’t use to exist. I had to find a way to be able to map all these fancy new capabilities into a simple category and checkbox. The decision that I made was this:- (1) All Categories and Projects will be “flattened” into a category. (2) All Project titles will have a prefix of “[P]”. (3) All incompatible items from Things will get mapped into a notes field.

(1) and (2) allows me to keep things simple, and (3) is simply how I handled iCal and Contacts. Keep in mind that Text::Unidecode and Encoding::FixLatin continues to provide the capability to help in the Chinese/English situation.

As in the To Do lists being generated from the CSV from things.sh, this was essentially a lot of file manipulation and string parsing. Nothing particularly interesting in the way the code was written in order to handle any oddities.

Memos

This was a little more complex than handling a Things CSV file. After some research, I have found that Memo syncing tends to be handled by individual text files. This works great for me as I’m all for text files. However, I’m an Evernote user and that’s definitely able to handle more than just text. So, a process has to be put in place for me to be able to get “Stuff” on my palm.

  1. I have to decide what I want in the Palm
  2. This has to be placed into a separate category in my Evernote. So a separate notebook is used e.g an Evernote Notebook called “ForPalm”
  3. I have to be able to get each individual note into a text file
  4. Convert the text file into a MemoDB file for Palm
  5. Sync that

I have to admit that the above has proven itself, over time, that it isn’t the best way to get memos into the unit, but it was for now the easiest. To be able to export the Evernote Notebook into a *.enex file requires a non-AppStore version of Evernote. Once the dump is obtained, I put it through a script I found called enex-dump. This small little tool in PHP helps to convert this dump into a folder of plain text documents. This folder is then parsed for content and transformed into the MemoDB file.

Buttons Functionality Complete

With this, the functionality of the basic 4 buttons of my palm is complete. Over time, small little fixes for display are found and fixed, but otherwise, everything is pretty stable. As long as I could export the data into the format required, be it ICS, VCard, CSV or text files, I have the capability of getting that information into my PalmIIIxe and show it.

All buttons mapped

Additional Notes

As the use of my Palm now works in tandem with my iPhone, i have noticed that To Dos and Memos tend to have the highest capacity for change. While i can hold off not syncing calendar items and contacts, To Dos and Memos then to require more immediate attention. And while the current mechanisms are 1-way sync, I’m starting to feel that there might be a need to change how I handle both these items such that 2-way sync can be achieved. I’m still weighing my options, it might require a new method, software architecture or perhaps some new way of addressing it. As with always, its a function of time I can spare for this project.

See Part 1, Part 2, Part 3

PalmIIIxe – Part 3 – Getting Contacts data into the Palm

If you have seen Part 1 and Part 2 of me trying to make use of the Palm IIIxe, this particular post would be no different. In fact, it might just be more of the same.

Contacts in Mac is not much different from iCal. We can essentially export all entries from the Contacts application into a vCard. This resulting *.vcf file can then be parsed and written into the Addressbook PDB format using PERL.

So whats different?

The key difference between Datebk and Address is (1) the availability of “categories” in Addresses, and (2) the higher likelihood of using Unicode characters. Coming from a multi-lingual society where most text are not ASCII, this was suddenly an interesting problem when parsing information in PERL and “depositing” the data into a device that is essentially ASCII only.

So… Categories

Handling categories was relatively simple. We just had to ensure that we were able to obtain the category information from the vCards and match/write them out into the same category IDs as in the PDB. If we can’t find a category entry in the vCard, then we just slotted it into “Unfiled”. The PERL libraries used were Text::vCard::Addressbook, and writing out to the address book PDB format using Palm::Address.

One other thing to note is that there are lots of items in the modern vCard format that didn’t used to be available in the Palm. The simple way to deal with all these was to just have ALL the information parsed and stored into the Notes field in the Palm. That way, I effectively don’t lose any information when looking up data, but also am able to just sort and keep the most important information visible.

Imported vCards in AddressBook and Categories Kept
Notes field keeping all vCard information intact

As with iCal, this is really a 1-way sync. Changes in the Palm wouldn’t be updated back into Contacts, but IMHO, that isn’t really required.

What about Unicode and UTF8?

This was the part that was interesting. Initially, parsing a small subset of my vCard export , handling categories and notes was successful. But when I finally exported the full data, the Palm crashed. Actually, I can’t really remember if it crashed or not, but I do remember that it didn’t go well. (*I’m writing this about 3/4 of a year I wrote the code). When I finally discovered what was wrong, it was clearly the fact that as awesome as the Palm Pilots were, it didn’t handle UTF8 too well. I had two solutions. (1) Install CJKOS as a way to get the right fonts on my device, and then get the data into the PDB using the right fonts, or (2) erase, negate the UTF8 data so that I would leave behind only the ASCII text.

Both weren’t particularly good solutions as (1) would result in me trying to ensure that the character sets are in sync all the time. Additionally, I couldn’t find CJKOS anymore. (2) was worse. I would lose data. This rendered some of the card information useless, as there are folks whom I only and have information in their native language. I have also gotten used to seeing their names in mandarin. Romanizing their names would therefore not be a good solution.

As I dug around for a solution, I finally came across a PERL module called Text::Unidecode. This seems to be an awesome little module that is able to, in the best of its ability, translate, phonetically, the words in Unicode, into its English, ASCII equivalent. And I’ll say, the results are pretty amazing. Testing on mandarin, it was able to get the Han Yu Pin Yin pronunciations of words I threw at it. Below is one of the examples.

Text::Unidecode handling the conversion of Unicode into Han Yu Pin Yin

I was impressed.

I immediately got this integrated into every field that I had across iCal and Contacts, and potentially saved me a lot of grief porting data across a modern system and a more vintage technology like the Palm.. which doesn’t understand Unicode.

With two functions down, I only have two more to go. Things and Evernote.

Update 14/1/2019: It seems that I have just came across a small bug that caused Unicode characters to be recognized as gibberish. This seems to be due to some kind of bad encoding on the text string, which my Mac is able to decode, but Text::Unidecode is unable to handle. In order to fix this, all strings are now passed through Encoding::FixLatin in order to ensure that the text is encoding correctly. Once done, Text::Unidecode performs as expected.

See Part 1, Part 2

PalmIIIxe – Part 2 – Getting iCal data into the Palm

Getting data into the Palm IIIxe is a challenge. First of all, it doesn’t have Wifi. Secondly, it uses its own data format *.PDB to store data. Which means not only do I have to find a way to sync the data over the serial port, I also have to be able to convert the data that I currently have into the compatible format.

As mentioned before in the previous post, pilot-xfer is my friend. It can help me sync data in a few ways into the Palm. Serial or network. Thank goodness that Palm at least had the sense to include Network syncing, but the pilot-link suite of tools is the one that made it possible to redirect a serial port to a network-ed system running a HotSync capable machine.

Diagram of how my Palm syncs data

Once I had this setup, getting files in and out of my Palm was possible. Not easy, but definitely possible. As shared before, the productivity utilities that I use daily are as follows.

  • iCal -> Datebk
  • Contacts -> Addressbook
  • ToDo -> Things3.0
  • Memo -> Evernote

This means that somehow, I have to convert the data from these systems into the respective PDB files that can be understood by the Palm built-in apps. I started off with iCal and Contacts as these have standard formats that we can convert into and thus transform into records that can then be written out into PDB. I’m familiar with PERL, so I started hunting for PERL packages that would handle Palm PDB files. Exporting from iCal and Contacts on Mac is easy, so there isn’t much to worry about there.

Handling ICS to Datebk PDB conversion

The PERL package I used falls under the PALM package and the ICS file parse I used is iCal::Parser. A few key things to note. What PALM can support is definitely simpler compared to what iCal can handle. So it is very important to NOT lose any data during the transformation. Therefore, all un-mappable information would go into the NOTES field in the PalmPilot. Its also important to know that the iCal Parser that I am using handles all the conversion of ICS entries into parsable calendar objects. This makes it much easier for me to be able to not worry about things like repeated events etc and just focus on the mapping. This isn’t a coding tutorial, so I wouldn’t be providing code to show how I got it done. But ultimately, I made some decisions about how things should show up on Datebk, manipulated the strings (PERL is good for things like that), provide by the parser, and wrote it out into a PDB.

High-level prices flow of the data conversion

Needless to say, it took a while for me to settle on exactly how I would like the data to show up and the abbreviations I would use, but the end result, I think, is pretty concise and usable. The screen, being only 160×160 also mean I had to choose carefully what I wanted to show. It really drives home the need to only show exactly what is required and important, and how far technology has come.

Datebk showing my iCal data
Notes field showing all other information meaningful to the event so that nothing is lost

Once the PDB file has been constructed, the next thing was to be able to move the file seamlessly to the networked HotSync machine for syncing. Making use of Dropbox to do this was no a no brainer. I previously had some Linux code that I wrote to be able to sync files, so I effectively just re-used those. I could have just used rclone to do the same. The flow below shows how the PDB file would move from the moment it has been constructed. As the files are very small, the cloud sync only really takes a moment.

Moving the file so that we can HotSync it over the network

Once the file is on the HotSync system, pilot-xfer and pilot-nredir does the rest.

Calendar Week view on iCal
View of the same week on Palm Datebk

Tada! 🙂
I now have my iCal available on my Palm.

Final bits on the conversion

Putting on the finish touches on this conversion process just requires me to now be able to automatically export the iCal entries using Apple Script, for all my calendars into a file, so that parsing script can then take over the conversion. It was at this point that I ran into some slight issues. I realised that I didn’t want to have my Mac handling the ICS file export and then running the conversion. This is largely because I didn’t want to have to re-install all those linux tools every time I change or upgrade my Mac. I wanted the tools to be on an OS whose tools are generally available as part of the system. I also wanted to ensure that all the other tools I ended up creating could follow the same structure. So I tweaked the process just a little bit….

Final ics file to Datebk syncing workflow

At the end of this, I could export any parsable file into a target folder in DropBox, and it will be MD5 checked for any changes. Once a change is detected, it would then automatically (re)create an updated PDB file which would be placed in a HotSync folder. Connecting my Palm to the RaspberryPi, which has pilot-nredir always running in a while loop, would then automatically sync, install the latest PDB file into it.

Issues?

Not much… everything I have seems to translate pretty well into text. The one thing that I have to remember that this sync is ONE-WAY. It gets iCal stuff into the Palm, but not any changes in Palm back into iCal. This means that I can’t create, edit, delete entries on the Palm and expect it to carry across to my iCal.

HOWEVER, I find that short-coming ok. I use my iPhone to keep track of my calendars anyway. The Palm is really just a way for me to not have to whip out my phone just to check a calendar entry if I’m playing games on it.

At some point in the future, I might have to re-design the entire process if I want to be able to do a 2-way sync, but it might not be worth the trouble.

See Part 1