Getting a good clean signal over the serial port was really important. After a fair bit of use of the acrylic laser cut connect, it was found that it constituted too much trial and error in manually making the connection. So the idea to actually modify the Palm serial port into a connector started to take root. The key idea is to look for a space on the Palm, solder the connector and have that connected permanently on the Palm, so that when required, an add-on board can be used to expand the capability of the unit, instead of mucking around with the 10-pin port.
Opening the Palm, Soldering the pins
Opening the palm was easy. I followed a post from iFixit to get that done. It was then decided to put the pin connectors on the flat portion at the back of the Palm. The PCB at that area was also empty giving me space to work the wires. I used an isolating tape to mask off the areas where the 10-pin was frequently used by external accessories and soldered “backward” the wires that would extend to the connecting pins.
The wires are then routed internally along the edge gaps of the palm and soldered underneath the cover.
On the PCB, insulating tape was also used on the PCB to prevent short circuits just incase the pins were to touch and scratch the board.
All these are then carefully re-assembled, hiding all the wires within the Palm. The end result is a neat package with no visible modifications to the 10-pin port.
The exposed pins shows up on the back, providing a slight improvement in viewing angle when placed flat on a table top. A cover was made to provide some amount of protection to the pins, but I have found that is usually unnecessary.
Upon re-assembly, a functional check was done to ensure that everything is still working…. and it is.. yeah!
Connecting an Expansion Board
The connector pins lines itself up to a 5x7cm DIY PCB. The connection is fairly straightforward and acts as a breakout board for additional functions like the adding a Hotsync button. Jumper wires are used to extend and connect the various pins to additional circuitry making it a lot neater to deal with. Some pictures below to illustrate what was done. Overall, it wasn’t rocket science, but it surely made the connections a lot easier to deal with.
All the above and previous work finally made sure that I was able to have a good way to achieve a “break-out” of the palm pilot pins unto a PCB which I can then work on. This made it a lot simpler to connect things like resistors and other ICs. As noted in Part 1, this kinda fixed issue 3-6. But now I have to begin working on question (3)…. how do I get the baud rate past 19200? And why can’t I get PPP working? All that experimentation would be detailed in another post.
Serial ports are obsolete. So trying to sync the PalmIII to a modern desktop becomes a pain when you need to convert the serial port to a USB. The use of an adapter is to unwieldy to carry about, so the hunt for a PalmIII serial to USB cable begins…. and it ends rather quickly. It is very hard to find, and very expensive. So comes the consideration to make one yourself. Keep in mind that what I wanted to do was to create a cheap Palm to USB serial connector so that I could have PalmOS talk to a linux… for the purpose of… whatever.
On browsing the internet, there is no lack of information when it comes to documenting the PIN outs of the PalmIII serial port. I have reproduced the one that I found the most simple/useful here.
This is just great, as it also seems to coincide with most of what other people have done with their old PalmIIIs, using it as a serial terminal. A lot of the mods out there seem to simply open up the Palm and solder stuff directly to the TX and RX lines and hooking it up to their Raspberry PI or otherwise. I didn’t want to do that as I didn’t want to “destroy” my Palm. I still wanted to be able to use it for games and to bring it around (thats what I wrote all the sync software for isn’t it?). So the goal was to have a way that my palm can interface with a modern linux system, over USB, get on the internet etc, so that I do fun stuff with it, and get continue to get the same goodness of the original Palm for use as a PDA.
This is where I made my first mistake.
Mistake 1: Not understanding Serial
I have mistakenly thought that it is simply enough to connect the TX/RX lines. But it turns out that TTL levels are very different from RS232 on a DB9 connector. The former ranges from 0-5V and the latter can swing from -9V to +9V or more. Even though both are serial ports. Hooking up directly shows gibberish on PocketTerm (by Plushworks), but it definitely shows something. This means that the first thing I had to do was to try and convert the RS232 signal to TTL. Upon digging a little more, I came across a connector for Serial to TTL. So I ordered one to try.
Problem 1: The Palm Serial Connector
In order to be able to create a direct connection from my PalmIII Hotsync Cable to TTL Serial in a small form, and do away with bulky connectors, I had to desolder the DB9, and directly interface my Palm connector to the serial port. Thats where the connection diagram proved useful. However, I didn’t want to destroy my Palm cable by cutting it. So I had to have a sustainable way to re-create the Hotsync connector so I could build things from it in the future. For that, I designed, with some interactions, an acrylic based connector which I can laser cut, then hook up at low cost. Its not fantastic, but it works.
There are a few issues however, that the height of the pins has to be exact. Otherwise the connection to the palm is flaky. So while doing this helped me get past certain roadblocks, using these connectors wasn’t the way I ultimately ended up with. More on that another time.
Hooking things up
Once I had the connector made, with breakout for all the lines, I followed the diagram below and started connecting it to the RS232 to TTL serial adapter. Putting everything on a breadboard, I decided to go a step further and actually put in some LEDs in order to have some visuals when there is traffic flowing or when the port is open.
The sketch below shows how I hooked up only 4 lines to the TTL and had my first successful sync with pilot-xfer. The connecting system is a Cubietruck which has an onboard TTL serial (UART0) which I connected to. The resulting port on the UART is /dev/ttyS0. The testing software used was, on the Palm side PocketTerm, and on the Linux side, Minicom. The settings used to test are documented in the sketch below.
The photo below shows the hardware build and connectivity on the PalmIII side. Notice that I have a breakout board on the Palm end with some LEDs which I subsequently added to help my visually identify the state of the serial connection. So this allows me to quickly identify connection behaviour and problems across the Palm the TTL conversion. I also added some connector pins so that I can use jumper wires to hook things into the breadboard without having to re-solder all the time. Another video below shows the process of a Hotsync…. just some blinking lights there. But I was happy that this was working.
Further testing with login services on Linux
In general, to be able to offer any kind of service over the serial port, it must not be “taken up” by any other service. So I had to ensure that there are no other boot up service that is using it. As I was using a Cubietruck with the Armbian image, I just had to disable the console from the boot environment.
On Cubietruck, Armbian Distribution delete “console” in armbianEnv.txt or change “console=display”
Just to bring it a step further, once I was able to establish a serial connection at 9600 or 19200 Baud to the Linux system, I tried offering login services on the Serial Port. In may posts, I have noticed that folks are trying all sorts of setup to be able to permanently offer a login prompt over serial. However, this is not something that I wish to do. I wanted to be able to (1) know exactly the service that I want to offer, and (2) have control over when I offer it. The following command provides a login prompt over the specific serial port.
On Cubietruck, Disable Serial Console on Boot cd /etc/systemd/system systemctl mask serial-getty@ttyS0.service sync reboot
Once done, you should be able to get to a login prompt when connecting over the serial line.
Problem 2: Maximum Baud Rate of 115200 is Unavailable
While everything works on 19200 on Minicom and PocketTerm, but when it comes to Hotsync, for some reason, I can’t get it to go above 19200 Baud rate. This holds true for Hotsync, but not the Minicom+PocketTerm combination. All research on the internet doesn’t seem to talk about why. This was a head scratcher for me. Every thing I have read seems to only say “4 lines i.e. Vcc, And, Tx, Rx, and it works”. Poking around the settings on the serial port in linux also didn’t seem to help. So while I was happy that I could now perform a Hotsync at 19200, I wasn’t completely happy as I couldn’t use the max rate of 115200.
Problem 3: Trying to Setup PPP from Palm to Linux does not work
Trying to survive on 19200 Baud rate, I tried to setup PPP on my linux system such that I would be able to get a network connection routed from my Palm to the internet via the linux machine. This was working using my bulky Serial to USB adapter, and I wanted to replicate this with my connection. However, this wasn’t the case. Trying to setup a connection continually fails, with the server side being stuck at the following message.
sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x7ba91055> <pcomp> <accomp>]
All this seems to indicate something was wrong. It seems that having just Tx and Rx lines being connected isn’t enough. I started to wonder if the RTS and CTS lines needs to be connected as well. And if DTR needed to be connected.
End of Part 1
At this stage, lets review what was accomplished/learnt/questions:- 1) I have managed to get my Palm connected to a UART TTL Serial Port at limited speed 2) A Max3232 logic convertor is required to convert RS232 signals to TTL 3) The base Palm Baud rate seems to be 19200. 4) Connecting with just TX and RX provides connectivity over serial at the base baud rate, but can’t seem to get to the max speed of 115200 for Networking or Hotsync 5) Do we need all 10 pins to be connected as is in the diagram to be able to replicate the functions of the original HotSync cable? 6) We need to have a clean, stable and extensible way of connecting the Palm Serial port to a breakout board for further development
Points 3-5 seems to be interrelated, and in order to solve it, I had to first have my serial connector be reliable as stated in point 6. As noted earlier, my self made connector seems a little flaky when trying to connect to the serial pins. There was therefore a need to change the way I connected to the serial pins before I tried to tackle this.
I think that, point (6), deserves another post. So till my next installation, where I will highlight what I did to improve the connectivity situation to the 10 pins on my Palm.
However, mundane the PalmPilot is in today’s modern landscape, its simplicity still intrigues me. So as I continue to try to document the various steps I have taken to bring back the technology to life, at least within my own sphere of use, the hope is that we can all get back to a point where technology really is an enabler rather than just a distraction. So here is how I handled the final two basic components offered in a Palm. To Dos and Memos.
I use Things 3.0 from Cultured Code to help sort my list of item that I need to complete. As the number of To Do lists out there are large and varied, I can’t say for sure if this method would work for anyone else unless you are also using something similar, or have a workflow similar to mine. The key idea is the same as the experience gathered from working on iCal and Contacts… To get the database exported to some common format so that I can work on it with the PERL Palm modules.
The key ingredient is this project available on Github called things.sh. This very capable command is able to get all your cloud connected Things information into a CSV file which you can then parse. There isn’t really a lot of fancy magic here. However, there are a few text manipulation issues we have to deal with.
Long names and spaces
Projects and Categories
These items above are just so common in modern lists that didn’t use to exist. I had to find a way to be able to map all these fancy new capabilities into a simple category and checkbox. The decision that I made was this:- (1) All Categories and Projects will be “flattened” into a category. (2) All Project titles will have a prefix of “[P]”. (3) All incompatible items from Things will get mapped into a notes field.
(1) and (2) allows me to keep things simple, and (3) is simply how I handled iCal and Contacts. Keep in mind that Text::Unidecode and Encoding::FixLatin continues to provide the capability to help in the Chinese/English situation.
As in the To Do lists being generated from the CSV from things.sh, this was essentially a lot of file manipulation and string parsing. Nothing particularly interesting in the way the code was written in order to handle any oddities.
This was a little more complex than handling a Things CSV file. After some research, I have found that Memo syncing tends to be handled by individual text files. This works great for me as I’m all for text files. However, I’m an Evernote user and that’s definitely able to handle more than just text. So, a process has to be put in place for me to be able to get “Stuff” on my palm.
I have to decide what I want in the Palm
This has to be placed into a separate category in my Evernote. So a separate notebook is used e.g an Evernote Notebook called “ForPalm”
I have to be able to get each individual note into a text file
Convert the text file into a MemoDB file for Palm
I have to admit that the above has proven itself, over time, that it isn’t the best way to get memos into the unit, but it was for now the easiest. To be able to export the Evernote Notebook into a *.enex file requires a non-AppStore version of Evernote. Once the dump is obtained, I put it through a script I found called enex-dump. This small little tool in PHP helps to convert this dump into a folder of plain text documents. This folder is then parsed for content and transformed into the MemoDB file.
Buttons Functionality Complete
With this, the functionality of the basic 4 buttons of my palm is complete. Over time, small little fixes for display are found and fixed, but otherwise, everything is pretty stable. As long as I could export the data into the format required, be it ICS, VCard, CSV or text files, I have the capability of getting that information into my PalmIIIxe and show it.
As the use of my Palm now works in tandem with my iPhone, i have noticed that To Dos and Memos tend to have the highest capacity for change. While i can hold off not syncing calendar items and contacts, To Dos and Memos then to require more immediate attention. And while the current mechanisms are 1-way sync, I’m starting to feel that there might be a need to change how I handle both these items such that 2-way sync can be achieved. I’m still weighing my options, it might require a new method, software architecture or perhaps some new way of addressing it. As with always, its a function of time I can spare for this project.
If you have seen Part 1 and Part 2 of me trying to make use of the Palm IIIxe, this particular post would be no different. In fact, it might just be more of the same.
Contacts in Mac is not much different from iCal. We can essentially export all entries from the Contacts application into a vCard. This resulting *.vcf file can then be parsed and written into the Addressbook PDB format using PERL.
So whats different?
The key difference between Datebk and Address is (1) the availability of “categories” in Addresses, and (2) the higher likelihood of using Unicode characters. Coming from a multi-lingual society where most text are not ASCII, this was suddenly an interesting problem when parsing information in PERL and “depositing” the data into a device that is essentially ASCII only.
Handling categories was relatively simple. We just had to ensure that we were able to obtain the category information from the vCards and match/write them out into the same category IDs as in the PDB. If we can’t find a category entry in the vCard, then we just slotted it into “Unfiled”. The PERL libraries used were Text::vCard::Addressbook, and writing out to the address book PDB format using Palm::Address.
One other thing to note is that there are lots of items in the modern vCard format that didn’t used to be available in the Palm. The simple way to deal with all these was to just have ALL the information parsed and stored into the Notes field in the Palm. That way, I effectively don’t lose any information when looking up data, but also am able to just sort and keep the most important information visible.
As with iCal, this is really a 1-way sync. Changes in the Palm wouldn’t be updated back into Contacts, but IMHO, that isn’t really required.
What about Unicode and UTF8?
This was the part that was interesting. Initially, parsing a small subset of my vCard export , handling categories and notes was successful. But when I finally exported the full data, the Palm crashed. Actually, I can’t really remember if it crashed or not, but I do remember that it didn’t go well. (*I’m writing this about 3/4 of a year I wrote the code). When I finally discovered what was wrong, it was clearly the fact that as awesome as the Palm Pilots were, it didn’t handle UTF8 too well. I had two solutions. (1) Install CJKOS as a way to get the right fonts on my device, and then get the data into the PDB using the right fonts, or (2) erase, negate the UTF8 data so that I would leave behind only the ASCII text.
Both weren’t particularly good solutions as (1) would result in me trying to ensure that the character sets are in sync all the time. Additionally, I couldn’t find CJKOS anymore. (2) was worse. I would lose data. This rendered some of the card information useless, as there are folks whom I only and have information in their native language. I have also gotten used to seeing their names in mandarin. Romanizing their names would therefore not be a good solution.
As I dug around for a solution, I finally came across a PERL module called Text::Unidecode. This seems to be an awesome little module that is able to, in the best of its ability, translate, phonetically, the words in Unicode, into its English, ASCII equivalent. And I’ll say, the results are pretty amazing. Testing on mandarin, it was able to get the Han Yu Pin Yin pronunciations of words I threw at it. Below is one of the examples.
I was impressed.
I immediately got this integrated into every field that I had across iCal and Contacts, and potentially saved me a lot of grief porting data across a modern system and a more vintage technology like the Palm.. which doesn’t understand Unicode.
With two functions down, I only have two more to go. Things and Evernote.
Update 14/1/2019: It seems that I have just came across a small bug that caused Unicode characters to be recognized as gibberish. This seems to be due to some kind of bad encoding on the text string, which my Mac is able to decode, but Text::Unidecode is unable to handle. In order to fix this, all strings are now passed through Encoding::FixLatin in order to ensure that the text is encoding correctly. Once done, Text::Unidecode performs as expected.