Chibi-Mikuvan: Getting “Tired” of All This; Making the HCL Actually Function; Plus Other Work

Monday, July 14th, 2014 @ 15:48 | Chibi-mikuvan

As the Detroit Maker Faire and its attendant PPPRS race draw closer, Chibi-Mikuvan testing and modifications have been intensifying still! With each silly night testing video (when the parking lot is empty) comes more rework and rethought of the original design elements. I’m actually glad it’s all being done now, and things are being broken during testing now, because there are a handful of things on the design which are turning out to be bad ideas suboptimal for reasons like ease of maintenance, parts swapping, etc.

The story continues where it last left off: with my indoor testing of the Hysterical Current Limiter. I was gradually pushing the maximum current higher and higher and driving around while staring at the scope screen to get a handle on what kind of surge currents it was seeing with the error of the controller being quite large at low speeds. With the limiting code itself working, I began to implement the ‘time decay’ where the controller would keep track of how long it’s been running over a designated continuous maximum current, and then slowly lower the maximum allowed current over a few seconds to the continuous level. The idea of this would be to get more acceleration ‘punch’ but not risk blowing the fuse.

The way this code worked was that it counted how many loops (fixed delta-T, hence how many seconds) the current stayed above the continuous limit, and obtained a maximum current based on a lookup table with the delta-t count as the index. It would reset if the current fell back under the continuous limit. Seems simple, right?

//Obtain a current limit value for this loop
    if(delta_ts_overcurrent > 512) local_max_amps = MAX_AMPS_CONT;
    else {
      //an 500+ entry table would be overkill, so here we change the max current ceiling every 8 loops (16 milliseconds)
      //This requires only 64 entries [0-63].
      local_max_amps = MAX_AMPS[(delta_ts_overcurrent / 8)];
    }

    //Hysterical current limit algorithm
    //Unlike in Ragebridge, we only care about driving current, so the case of decrementing throttle quickly is not handled.
    if((amps_to_use > (float)local_max_amps) && limited_current) { 
      //local_max_amps = Either a fixed constant or obtained from a LUT
      throut -= OC_MULTIPLIER * RAMP_US_PER_LOOP;
      if(throut <= 1500) throut = 1500; //Don't go past neutral
      delta_ts_overcurrent++;
    } else {
      delta_ts_overcurrent = 0;
    }

That’s where the logic was flawed. Recall that the current waveform is messy. It’s a noisy sawtooth, not a smooth monotonic function. As I found out during testing, the chances that the current waveform goes under the MAX_AMPS_CONT value and therefore resets the lookup table index was almost 100% – when it worked, it was a bug.

I finally pushed it a little too far when doing a 120 amp launch (which could have seen errors as high as 200 amps, knowing the low speed performance). The current trace never sank, and I managed to mentally process that this was probably bad a few milliseconds before the fuse went out.

On dumb DC systems, a fuse-out will just cause everything to stop working. But this controller has regeneration ability – and for whatever reason, the failure mode of loss of battery input is “snap to 100% regen”. If the battery were present, a large burst of negative current will flow (into the battery) and I will faceplant through the scope screen. But without the battery there, regenerating into space just causes extremely high voltage spikes. I could smell the magic smoke coming from the ammo box right after stopping.

Fortunately, only the controller itself was toast. After resetting the power, everything else turned back on and still worked. Unfortunately, digging the controller out of the box is a 15+ minute service job. In the field, this was going to be intolerable. Furthermore, I now had a serious problem. A fuse-out is guaranteed controller death, so I have to keep the overload bursts conservative with the consequence of reduced acceleration.

I first turned my attention to the code. It took me a while of thinking to recognize the problem I described with it, and then a bit more thinking about how to fix it.

//Hysterical current limit algorithm
    //Unlike in Ragebridge, we only care about driving current, so the case of decrementing throttle quickly is not handled.
    if(limited_current) {
      if((amps_to_use > (float)local_max_amps)) { 
        //local_max_amps is either a fixed constant or obtained from a LUT
        throut -= OC_MULTIPLIER * RAMP_US_PER_LOOP;

        //This causes the time-current curve to only reset if the current flow goes UNDER the continuous limit.
        if(amps_to_use > MAX_AMPS_CONT) delta_ts_overcurrent++; 
        //Don't go past neutral
        if(throut <= 1500) throut = 1500; 

      } else {
        //Even if we don't limit the throttle this go-around, only reset the time-current decay counter IF
        //the current has fallen under the prescribed limit, indicating you've let off the throttle
        //or have used the regen brake 
        if(amps_to_use < NOLOAD_AMPS) delta_ts_overcurrent = 0;
        else delta_ts_overcurrent++; 
      }
    }

Here is the new logic structure. A new variable has been added – NOLOAD_AMPS. This isn’t a “no load” current in the motor equation sense but rather a lower limit to the current limiter – at what point does it turn off and reset? Now, the delta_ts_overcurrent counter will continue incrementing as long as the current does not fall under this lower bound. I set NOLOAD_AMPS at reasonably low number to start – 20 amps. This is the correct implementation of the behavior I described firstly – that the limit will reset essentially only if I let the throttle off in the process of being about to brake, or after the vehicle’s reached cruising speed anyway.

(Notice the flag limited_current up to… You know what this means).

With the code fixed, now I actually had to fix the thing. I decided to implement a change that I had thought about since the Trackstar failed the first time due to its signal board falling off: moving it ouside, right next to the motor. This has the effect of creating a single swappable module that consisted of the gearbox, motor, and controller, so if something goes sideways in the race, it can be swapped with a spare entire unit in a minute and then someone else can take their sweet time fixing the failed unit later.

I designed a new motor mounting plate that has an added ‘tab’ where the controller sits.

…and cut it out on my mechanical waterjet. Wait, what? Yeah, I went back to using the Shopbot gantry router as a makeshift waterjet with an 1/8″ wide nozzle. There’s nothing really stopping this machine from cutting metal, but its limitations are twofold – one, it isn’t nearly as rigid as a CNC mill, and two, it doesn’t have any provisions for chip removal or coolant.

The solution to the former is to take light passes at aggressive feedrates – keep the tool cutting, but make the cutting forces tiny. I whipped out my old experimental setting from the last time I did this, and refined it a bit more. The final settings I used were 18,000 RPM, 125 inches per minute, and 0.025″ deep passes.

The solution to the latter is more complicated. Coated carbide endmills are great about not having build-up problems, but chip recutting (chips being sucked into the next pass) will easily kill the little 1/8″ endmills from past experience. I used a dual approach of slathering the area with Tap Magic Aluminum cutting fluid as well as keeping a compressor air gun handy to clear chips on every pass. I’m not as confident machining aluminum dry, because even a little bit of build-up can break the cutter – at the least, I think it needs a mist coolant system where a tiny bit of coolant is dropped into an air blast stream.

On the whole, it worked great. Here’s the tab bent into place after the holes were cleaned up. Mechanical waterjet!

And here is one of the spare Trackstars installed onto the tab. This is now the unit that I’ll make a spare(s) of and replace whole in the event of chaos. The Trackstar is also in a much better cooling position here, being in the airstream.

As I was extending the power cable coming from the contactor inside the ammo box, I had the idea of Adding More Buscap. Galactic amounts of bulk capacitance helps smooth the DC input rails by isolating peak current spikes from the battery (and hence current sensor) – this should help clean up the readings some as well as supply the controller with more stable power. I basically tripled the mount of capacitance for the controller here, with four 820uF capacitors. The 8 gauge short wire leads shouldn’t cause trouble since the ESR of the capacitors should still dominate the amount of current ripple through them, but ideally you’d want them as close to the controller as possible. This shit’s important – the inside of high powered motor controllers are basically made of capacitors.

The caps were bundled, soldered together, then the soldered legs bent over to attach to the wire. It was wrapped up in a heat shrink burrito.

With this setup, I verified that the ‘time decay’ element of the current limiter was indeed working, and we went out once more. The idea of these testing rounds was to get everyone on the team who might be driving (and we seem to have JUST enough people to have drivers, a pit money, and an event staff helper) to get some driving time in WHILE possibly breaking it so we can fix it ASAP.

The results of this test were fantastic – the current limit was still set rather conservatively, there were no fuse problems, and I think people are realizing what they got themselves into. The noise coming out of the gearbox at the end was indicative of future troubles. That test actually ended with the tapered input shaft of the gearbox pinion slipping once again – I came into the near turnaround with full regenerative braking, then suddenly the gearbox no longer output torque.

This was to be the last time the tapered shaft will bother me, because I’m done with this 1,1,1, Trichloroethane shit.

The spare angle grinders I bought (these) have input shafts with Woodruff keys instead of…. well, nothing.

I decided to prepare these for use in the vehicle and only keep the tapered-shaft gearbox around for emergency spares if all else has failed. The stock Woodruff key seemed to be a 3/8″ or 10mm one, which is a bit small for my tastes, so I bought some 1/2″ circle ones to upgrade the size.

I took a chunk of 12mm shaft and turned one end down to 10mm to mate with the bore of the gear…

I borrowed a little woodruff cutter to make the slot in the side.

The completed new input shaft assembly. This time, it even installs from the outside of the gearbox, so I no longer have to open it up to remove the input shaft. On the previous Harbor Freight gearbox, the plate holding the bearing in came with pre-stripped screws, so I couldn’t remove it. On this one, they were removable, but low quality, so I replaced them with M4 socket cap screws.

I did repair the previous tapered shaft gearbox. This time, I took care to clean up the taper where it had slipped in both gear and shaft. Then I drilled out the screws I couldn’t remove from the bearing retainer plate, freeing it up and allowing this gearbox to also be serviced from the outside. Then I put the gear and shaft on a 5 ton press and shoved until it could move no more. This gearbox will now be on reserve as the last-ditch backup.

I also finally made a lower battery bracket to prevent the half-sized battery from pivoting on the frame because it only has two coaxial fastening locations. This makes battery changes way, way easier.

With these modifications done, we had the most insane testing session so far:

The goal of this testing round was to stress the new gearbox as much as possible. I also wanted to see how adding the shell affected the handling, so at the risk of putting another dent into it, we brought it along. The good news is that it doesn’t much – since it weighs so little, driver weight differences still dominate (Time to diet!).

However, this was when the weakness of the Harbor Freight pink wheels finally showed. I’ve had this set since the thing was built, and the rears were definitely showing wear, but little did I know how thin the tread area actually was. Ben, the test driver of the last lap, pulled probably the best save I have seen yet when the rear right tire blew out on a turn.

I may have died on the spot if it planted into the curbside.

I cut open the failed tire and noted how thin the tread was. Even if the tread were new, there would only be about 3mm or so of it surrounding the tube, which seemed very low and could result in the tire being fragile. Its also inconsistently molded, as you can see from some spots being hair-thin (around the failed location) and some spots still having thick tread.

So I swapped on another set of pink wheels with the goal of driving the tires off. We would purposefully push hard, drift, and slide, just to see a worst case tire wear scenario:

Here’s that video.

One thing we began favoring was using the powerful regenerative braking on the rear axle along with weight transfer to help kick the rear end around. I figure this wouldn’t be very useful except in certain cases of executing a pass on someone’s inside, but it was one way to get some artificial wear and tear on the tires and also beat them up. Once again, Ben is the destructor of Harbor Freight tires.

This was an eye-opening test. Even though that happened at the very end, after 14 of those S-curve laps and 15 loops around the outside, it began turning me off to the Harbor Freight 8″ wheels. They seemed okay for low speeds, but the tread was too thin to be durable otherwise.

I also don’t think they’re vulcanized, or if they are, they’re not done well. This was the wear pattern on the tread of the outside front wheel after that test – the rest weren’t much better, and all showed this “feathering” wear. After this test, I was now totally dissatisfied with the Harbor Freight pink wheels – they’re nice and grippy, but the tread life on them is extremely limited due to how thin the tread is.

Other options for tires in this “handtruck” size were limited. It’s smaller than the most common small industrial tire (4.10/3.50-4) that more manufacturers make. I ordered some of Northern Tool 189399, and since it comes from an actual known tire/wheel company, they’ll hopefully be more durable.

In the mean time, I swapped to a set of Cheng Shin 2.50-4 tires that I bought from Surplus Center (but whose page is no longer up). The difference between these and even a new Harbor Freight pink tire was striking – the tread area was far more rigid to a finger push than the HF wheels which just deformed inwards. The sidewalls were also much stiffer. I only have 6 of these tires, however, so I’m countting on the Northern Tool ones being legitimate.

Finally, we went to fool around on these, but only recorded the shenanigans:

Overall, I’m much more satisfied with these – a couple laps of hard driving and they hardly looked different.

While awaiting the order from Northern Tool, I’m going to take care of one very irritating problem that made itself painfully noticeable during all this testing: Changing the tires. Right now, you have to thread a bolt through two rim halves AND have it line up with the hub’s threaded holes! I’m not sure why I didn’t go with a stud method like every single automotive hub in existence (Probably because I was like “Ooh, these wheels come with free bolts!”).

The plan to remedy this is just to thread the bolts in from the other side, tighten them, and then tack weld them to the hub, then use proper nuts on the new “studs”.

I think there will be more hard testing to come once I perform this bolt to stud conversion, and at least one “full dress rehearsal” involving a live battery and tire change. I’ve ordered a little aluminum hydraulic jack to assist in the process once I discovered how impossible it was to change the tires quickly.

 

Recently

  • Chibi-Mikuvan: Detroit Recap and The Conclusion
  • Chibi-Mikuvan: The Penultimate Update!
  • Chibi-Mikuvan: Getting “Tired” of All This; Making the HCL Actually Function; Plus Other Work
  • Chibi-Mikuvan: Experimenting with the Trackstar 200A and the Hysterical Current Limiter
  • Beyond Unboxing: The Great Cambridge Chainsaw Massacre; Ryobi RY40511 Cordless Chainsaw
  • Chibi-Mikuvan The Extended Universe Edition: Water Cooling Loop and Digital Dash
  • Chibi-Mikuvan: More Stickers is More Power, Right?
  • Miscellaneous Van Adventures: Fuel and Coolant
  • 2.00gokart: The 2014 Season!
  • The Turbulent Rise of Chibi-Mikuvan
  •  

    7 Responses to “Chibi-Mikuvan: Getting “Tired” of All This; Making the HCL Actually Function; Plus Other Work”

    1. Max Says:

      Well, we all have to learn at some point that the signals we get are not the ideal ones we learn about in school, but I’m not sure you chose the best solution to help that. With the disclaimer the my understanding of the electrics involving the motor tend to zero, it seems to me you just traded a “reset if below a limit” condition to a… “reset if below some lower limit” one. Perhaps you _know_ there will never be erroneous spikes below _that_ limit, but I prefer the “filter your inputs first, act on them only after that” approach, regardless of how exactly that gets implemented in code. As in, a single reading of anything should never be able to effect anything…

    2. the chuxxor Says:

      This situation is a little different from measurement noise, since the waveform is supposed to be sawtoothy – but I forgot about this fact. Ideally, yeah I’d be able to sample many times and filter, but nothing about this control loop is remotely legitimate in the sense that the sensing resolution, output resolution, and sensing frequency are all technically too low for the demands.

      The lower limit is pretty much exactly that… it’s low enough that if the spikes are high enough in magnitude to cause it to reset then, things have already gone to shit!

    3. Okuu The Engineer Says:

      Have you considered using actual gokart tires, or am I showing ignorance of some ruleset?

    4. the chuxxor Says:

      Haha, yup, the PRS ruleset explicitly forbids go-kart slicks and racing tire compounds. Pretty much everyone runs handtruck tires and bike tires.

    5. Marshall Says:

      Have you considered putting a beefy diode in parallel with the fuse? (At least for testing if not allowed in the actual race) That way if it blows you still loose power, but the voltage spike is clamped to non magic smoke producing levels and the battery still absorbs the charge. But that’s just my two cents.

    6. the chuxxor Says:

      I have strongly considered this – not a diode in parallel with the fuse, since that would bypass the fuse in the direction it could protect in, but a large TVS diode at the controller that straddles the + and -. A 35v or 40v TVS would work well. It would avalanche and clamp the terminal voltage to a safe level if the condition occurred again.

    7. Hydron Says:

      I think you might struggle with solely a TVS type solution – if the controller tries to regen the full kinetic energy of the chibi-mikuvan into a TVS then it will see very large power dissipation and temperature rise. Another thing to remember is that the voltage across the clamping TVS will rise with increasing current – you’d need to size it to prevent this going above the controller rating.

      There may be other ways around the issue – e.g. keep the TVS but also monitor voltage across the fuse (should be ~0V until it blows) with an analogue circuit which shuts down the motor controller when the fuse pops. The TVS would catch any initial spike but wouldn’t have to deal with the kJs of kinetic energy of the vehicle if the controller lets the motor freewheel once the fuse goes.

      Another option may be to have a crowbar circuit with a big beefy SCR and a length of resistance wire which shorts out the input once the voltage gets too high, but this also has issues (high controller regen current into the near dead short, face slamming into steering bar with heavy regen braking force, limits on peak power dissipation in both resistors and SCRs requiring being careful that the crowbar circuit doesn’t also grenade). TVS again would also be useful for something like this to catch the initial spike. I’ve run into some of these issues designing a safety discharge circuit for a multi-kJ capacitor bank, and mikeselectricstuff has a youtube video about dummy load resistors with his own story about exploding resistors.

    Leave a Reply

    XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

    Prove you are human by reading this resistor:
    0Ω+/- 5%

    0
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9

    0
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9

    0
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9

    5
    5
    10
    20

    Match the sliders on the left to each color band on the resistor.

    Click Here for a new resistor image.

    If you'd like to learn more, read about resistor color codes here.