I'm at my wit's end here, hoping someone has run into something similar, or can look at the dump and see something that looks glaringly wrong.
Here's a trace dump of what I'm seeing:
xmit [A4][01][4A][00][EF] // reset
xmit [A4][09][46][00][XX][XX][XX][XX][XX][XX][XX][XX][21] // set network key
-> SetNetworkKey timed out
xmit [A4][03][42][00][10][00][F5] assign channel
recv [A4][03][40][00][42][00][A5]
xmit [A4][05][51][00][01][00][7A][01][8B] set channel id
-> SetChannelId timed out
xmit [A4][02][45][00][39][E3] // set channel freq
-> SetChannelRFFreq timed out
xmit [A4][03][43][00][A6][1F][42] // set channel period
-> SetChannelPeriod timed out
xmit [A4][01][4B][00][EE] // open channel
recv [A4][03][40][00][4B][18][B4]
the final recv line's status code (0x18) is CHANNEL_ID_NOT_SET, which is what I'd expect given that SetChannelId (and most other commands) times out.
after the reset, I'm waiting 1000ms, and for every command I wait a maximum of 500ms for an ack. I figured there might be something wrong with the serial connection, but I'm getting one valid response (from AssignChannel), as well as the error response for OpenChannel.
I can also post code if that would help.