Okay, Internet, tell me more about wireless bootloading.

One of my perennial bad electrical engineering habits is black-boxing electronics and software – not making much of an effort to understand how a (usually software) system works, just accepting that a solution is available and not extensively exploring it unless needed. Historically, I did this to electronic hardware too, coming from a more R/C hobbyist background (I don’t really consider the battlebots to be “robotics”) where things like “Receiver” and “Motor Controller” were considered atomic components as much as screws – they could not, and should not, be decomposed further. I’ve gotten a bit better at that, what with making entire motor controllers and all, but embedded programming is still one of my sour spots.

This is part of the reason why I like Arduino so much – it’s easy, reasonably well featured, and I don’t have to think about diddling registers and setting bits – 6.115 taught me to never do that again if I could avoid it – just to toggle an output pin. It makes the software experience just a little more like high level application programming, the kind I did before getting into hardware and actually all through high school (Did you know that I was CS before I was EE or ME?). Convenience is of the utmost importance, absolute code efficiency and running speed less so. Probably because I just haven’t done a complicated enough motor controller yet, or a flying thing with enough axes. Granted, I’ve also gotten slightly better at that, since there are just some things which are easier to do, like changing the PWM frequency of analogWrite(), by breezing through the manual and just doing the annoying bit shift thing, or enabling a regular interrupt so some part of my code runs with a fixed delta-time for control calculations. In summary, I favor trying and modifying existing solutions first in order to solve the high level problem of make thing work. Which is another great thing about Arduino and the general OSHW community – someone has probably done it already. Including build a smaller quadrotor, but that’s besides the point.

But I think I’ve reached the end of reasonable high level probing, because Chibicopter is not wanting to program wirelessly at all.

Here’s the rundown. I’m using the Adafruit Solution for wireless code uploading to the ATmega328-turned-arduino. There are some differences between my exact circuit and the “community solution” in this case – I’m running 3.0v logic power and my reset line is coming out of D6 instead of D3. The Little Purple Wire hacks are documented here. Otherwise, the level shifter/buffer circuit (which I feel isn’t really necessary) is the same, and for the first few attempts, the settings for my XBee Series 1 radios were also exactly the same as directed.

So here’s what’s been going on:

The Symptom: The remote reset always worked, but the program transmission itself has a success rate of maybe 5 to 10% at most. The most common symptom is hanging – the RSSI LED on the transmitting Xbee goes out, and the ‘Uploading” stage of the Arduino IDE never completes until it times out with an Out of Sync error. A few times, I’ve had programs make it all the way through and complete, but only once or twice, and then it becomes completely up to chance again. The successful attempts seemed to require uncorrelated “adjustments” such as exact location in room (possible radio bad spots or WiFi interference) and orientation of radio, which at such close range I find implausible to be the problem.

The fixes:

1. Complete hardware swap. I ordered more XBee Adapters because my one remaining unit was clearly showing reliability problems – I had to seat the XBee in the headers in a specific half-out fashion to get the power and associate LEDs to turn on, so to rule out the possibility of bad contacts or intermittent power, I made some new boards up. This did not result in any appreciable change in the operation – the vast majority of uploads still failed.

2. More radio power and XBee swap To test the interference theory, I borrowed Shane’s XBee Pro units which transmit at something like 150mW, which tends to outpower every 2.4ghz device around it. This did not affect the success rate of upload, but in this process one of my transmitting XBees was found out to be damaged. It was replaced with another of my radios (the one which had been handling LandBearShark’s load cell reading telemetry), to no appreciable change. I then borrowed two brand new radios, also to no appreciable difference.

3. Reduction to Tx/Rx only with stock XBee settings I removed the reset circuit’s digital I/O pin function so the pin did not toggle. The programming was then done just by hitting the reset button while uploading with only Tx and Rx active. This got me 2 or 3 consistent uploads (variations of Blink.ino), but after that, it once again become unreliable. I completely refreshed the settings a few times, changing only the baud rate to 57600 8-N-1.

4. Possible undervoltage operation problems The XBee is rated to operate between 2.8 and 3.5 volts. I’m running 3.0v, technically not really “3.3v”, so it could be that the XBee is being underpowered. Testing on real 3.3 volts provided by a bench power supply, using variations of Blink.ino only resulted in the discovery of the “uncorrelated adjustments” like Shane standing 5 feet to my left vs. next to my right, or using the back half of MITERS vs. the front, and turning my wireless LAN/Bluetooth off on my laptop vs. on. I do not believe any of those environmental factors are truly the cause of the problem.

5. Power Supply instability. Xbees, being high powered digital devices, draw current in bursts. I have very little 3.0v capacitance, so maybe little dips on the power supply was causing problems such as when the receiving end sent back start/stop flags. I piled a ridiculous amount of caps on the XBee pins directly during the above power supply runs, to no appreciable gain.

6. Trying a known working board. The 2.007 Nano carrier can wirelessly program using the “community solution” or using bone-stock settings on the XBees and manually pressing the reset button. This was demonstrated several times by Shane, but I could not get any of my hardware to reliably work with it either! This was eye opening, and pointed me to the fact that the problem may not lie with Chibicopter’s board. The full battery of tests – my Xbee adapter with both my radios and his radios, the XBee Pro, his own XBee USB dongle… none of them worked reliably if my computer was involved!

7. Messing with virtual COM port settings I tried the established solution (SET RTS ON CLOSE) with no other changes, changing the baud rate explicitly to 57600bps, and even installing a fresh, new FTDI cable to talk to the XBee adapter (thus installing an entirely new virtual COM with no possible previous changes). While the latter change appeared to result in success (4 reliable uploads), it stopped being reliable again thereafter. So promising ;-;

8. Literally trying wires. Hooking up transmitting TX to receiving RX, and vice versa, with GND bridged. This was literally plugging wires in and out of XBee sockets, and it worked without issue every time. This means the issue lies strictly between how the XBees interact with my computer and with eachother – once I literally bridged the connection, there were no communication problems.

Possible explanations we gathered:

1. Some hardware level difference between computers. While this seems to be a stretch, Shane’s Dell notebook can program Chibicopter with much higher reliability such that the random interference, voltage dip, and noise explanations begin to be plausible, and the Nano carrier essentially flawlessly. My HP DV7 can do neither. I’m not up to date on integrated computer peripheral hardware, but I do want to try the exact same parts on a Macbook – if it works on a Mac, then that would explain why there are so few complaints about it, given the Arduino and OSHW community demographic…

2. I am actually missing something very fundamental and stupid. Given that the uploads are so reliably unreliable, this seems almost like the most plausible explanation. Did I really forget to ground something? Are my traces actually soldered? They must be, because I can definitely program the damn thing over 3 wire serial! Are my XBees just cancerous? They’re both  Series 1 whip antenna types with the latest 10ED firmware now.

I’m willing to rerun some of the tests and provide more verbose error message logs, as well as COM port settings and XBee settings. But what the ass?







11 thoughts on “Okay, Internet, tell me more about wireless bootloading.”

  1. It sounds like a combination of 2 problems:
    1) Your board is unreliable.
    2) Your computer is causing problems (or the user is causing your computer to cause problems).

    If Shane’s computer can reliably program a known working board and your board some of the time, but your computer can’t program either reliably, it’s your computer (HW or SW or user). Get your computer (or a new one) to reliably program the 2.007 Nano board first, Then go look for problems on the Chibicopter’s board.


  2. I’d take a look at the serial data coming out of the FTDI cable with a scope, and compare it to that from the known working PC, just to see if there is any timing weirdness. USB->RS232 adaptors can sometimes cause strange issues when compared to a “real” serial port, ranging from subtle issues like slowness to plainly not working. As an example, one of the products I work on takes multiple minutes to program via a USB->RS232 cable under win XP, about a minute under win 7, and about 30 seconds using a “real” hardware serial port with either OS. In this case it seems to be due to the use of very small packets and acknowledgements in the programming protocol, which USB messes up by buffering rather than sending straight through, delaying the ACKs and thus slowing the whole thing down. If you can, I’d try using an expresscard->RS232 adaptor (make sure its not internally USB-RS232, as most are) to rule out a weird USB driver/OS/cable interaction.

  3. Just a quick additional note – if you do try a real port then you’ll likely need a MAX232 or similar level shifter, as I’m assuming that the FTDI cable you’re using currently has 3.3V I/Os rather than the standard RS232 levels (anything from +/- 3V to +/- 15V!).

  4. So wireless bootloading doesn’t work on BabyCorntroller either, despite using a really similar setup to Shane’s…

    I think the lesson here is that Shane is magical.

  5. “it seems to be due to the use of very small packets and acknowledgements in the programming protocol, which USB messes up by buffering”

    This is very much inline with my suspicions. It seems to be that there are either one too many, or too-small buffers. Something is being underrun, and this interrupts the program code transmission in the middle of a page, and the Arduino fucks up and aborts.

    I think Real Hardware Serial Port is possibly next on my list of things to try – if I don’t end up gluing a 6 pin FTDI header to the thing first!

  6. I’d need to get to one of my FTDI cables first, but I wonder if there is a way to change how the virtual COM buffers data. There are a few more hardware settings I have not yet messed with.

  7. Have you tried probing the data (with a logic analyzer or serial dongle) going out of your USBXBee and comparing it to what the ATmega sees?

    Also, is the transmitting XBee asserting !CTS ever? That would happen if the XBee’s data in buffer is about to be full.

    If you have buffer overruns anywhere it’s probably at the XBees, so maybe try having avrdude use hardware flow control?

  8. A coupe questions:

    how is the xbee powered on the computer side? From USB?

    Have you tried wireless programming another arduino? It looks like that’s exactly what you did with number 6.

    If that is the case, and the xbee is powered from USB and have no external power supply on the computer side, then your computer likely has underpowered USB ports and/or the XBEE exceeds the USB spec. Then likely the xbee is not getting enough power form your usb port. Got a USB powered 2.5″ portable hard drive? I bet that won’t work properly on your computer either. You could also check the XBEE documentation and see what the required power is. USB spec is 5V 500mA (though the device has to ask for power if it wants more than 100mA or so). If the XBEE is anywhere near 500mA (like 400+) then that is most likely your issue.

    easiest and cheapest solution: get a powered usb hub, or power the xbee externally.
    I guess you could also try a big cap on the usb power rail also (like maybe 2 470µf guys in parallel).

    And In my experience, Apple laptops have USB ports that are capable of sourcing enough current for that sort of thing, so it probably doesn’t have many problems on apple computers. Other computer brands are a lot more varied.

  9. You can try to lower the receive & transmit buffer sizes and the latency timer (to “1”) in the advanced port settings, but to be honest when push comes to shove, a single visual check comparing signals on a non-functional and a similar functional setup – even with a sampling oscilloscope if a logic analyzer is not at hand – is worth more than thousands of blind crackpot “restart/reinstall windows/drivers” style theories. Just sayin’.

Comments are closed.