Voltage changes are slooow

[cmock]

Hi,

I'm working on PID control for temperature ATM, and I find that changing the voltage is very slow, especially downwards -- e.g. changing from 4V to 1.8V takes about 1.5 seconds. Going up from 0 to 4V only takes about 0.2 seconds.

I've read #24 and understand it's not a good idea to change the CMR too quickly, but can it be more quick?

[ReservedField]

If you're working on PID you may want to talk to @s0be and check out TeamCloud's work, as they already have very good PID tunings and an almost-complete CFW.

Story time: OFW uses a 5kHz feedback. That's what SDK used at first. Then I went with 25kHz because it looked clean enough and had better slew rate (which is what you're talking about). Then we had spurious OPEN errors and I reduced to 2.5kHz as a workaround. Now I have fixed those errors (almost - need a windowed average on ADC cache to get rid of them). So now we could up it to 5 or 25.

Give me 10 mins and I'll post a patch for you to try.

[ReservedField]

This patch takes it up to 25kHz. I had to change the OPEN check slightly as it was triggering too much. It seems to behave well against vtc-VaporWare.

diff --git a/src/atomizer/Atomizer.c b/src/atomizer/Atomizer.c
index e2212bc..ce3652d 100755
--- a/src/atomizer/Atomizer.c
+++ b/src/atomizer/Atomizer.c
@@ -182,14 +182,14 @@ static volatile uint8_t Atomizer_forceMeasure;
 static volatile uint16_t Atomizer_tempTargetVolts;

 /**
- * Warmup timer counter. Each tick is 400us.
- * Counts up to 6 (2ms) and stops.
+ * Warmup timer counter. Each tick is 40us.
+ * Counts up to 51 (2ms) and stops.
  */
 static volatile uint8_t Atomizer_timerCountWarmup;

 /**
- * Refresh timer counter. Each tick is 400us.
- * Counts up to 500 (200ms) and stops.
+ * Refresh timer counter. Each tick is 40us.
+ * Counts up to 5000 (200ms) and stops.
  */
 static volatile uint16_t Atomizer_timerCountRefresh;

@@ -353,7 +353,7 @@ static void Atomizer_NegativeFeedback(uint32_t unused) {
        ADC_MODULE_VBAT, ADC_MODULE_TEMP
    }, 4, 0);

-   if(Atomizer_timerCountRefresh != 500) {
+   if(Atomizer_timerCountRefresh != 5000) {
        Atomizer_timerCountRefresh++;
    }
    else {
@@ -364,7 +364,7 @@ static void Atomizer_NegativeFeedback(uint32_t unused) {
        return;
    }

-   if(Atomizer_timerCountWarmup != 6) {
+   if(Atomizer_timerCountWarmup != 51) {
        Atomizer_timerCountWarmup++;
    }
    else {
@@ -384,13 +384,13 @@ static void Atomizer_NegativeFeedback(uint32_t unused) {
    else if(ATOMIZER_ADC_WEAKBATT(adcBattery)) {
        Atomizer_SetError(WEAK_BATT);
    }
-   else if(Atomizer_timerCountWarmup > 3) {
+   else if(Atomizer_timerCountWarmup > 25) {
        // Start checking resistance after ~1ms
        resistance = ATOMIZER_ADC_RESISTANCE(adcVoltage, adcCurrent);
        if(resistance >= 5 && resistance < 40) {
            Atomizer_SetError(SHORT);
        }
-       else if(adcVoltage != 0 && adcCurrent < 5) {
+       else if(adcVoltage > 30 && adcCurrent < 5) {
            Atomizer_SetError(OPEN);
        }
    }
@@ -525,10 +525,10 @@ void Atomizer_Init() {
    Atomizer_errorCallbackPtr = NULL;
    ATOMIZER_TIMER_RESET();

-   // Setup 2.5kHz timer for negative feedback cycle
+   // Setup 25kHz timer for negative feedback cycle
    // This function should run during system init, so
    // the user hasn't had time to create timers yet.
-   Timer_CreateTimer(2500, 1, Atomizer_NegativeFeedback, 0);
+   Timer_CreateTimer(25000, 1, Atomizer_NegativeFeedback, 0);
 }

 void Atomizer_SetOutputVoltage(uint16_t volts) {

Decreasing voltage is expected to be slower than increasing. This is because of the output capacitor. Some quick math from your measurements yields me -1.5V/s and +20V/s. Which means decreasing voltage is about 13 times slower. PID for atomizer doesn't look like a good idea, but maybe introducing some non-linearity in there could be.

[cmock]

WRT VaporWare: their PID tunings didn't work for my coil builds at all, which is to be expected. So I took my PID code, glued it into examples/atomizer, and started playing around with autotuning, which is my main point of interest (and when I get something, I plan on integrating it into VaporWare).

Thanks for the patch. but it seems it doesn't change a thing. I don't understand it, because AFAICT it should now be 10 times quicker, but it doesn't. This plot is pretty typical for the situation before and after the patch, see the second part where I had a puff from a hot coil, and while the red line (power requested by the PID) goes to zero, the green line (power as calculated from U*I) decays very slowly (left Y axis is in Watts; right Y and blue line are temperature; X axis is in 1/100 of a second).

What I also don't understand is why decreasing voltage is slower -- from the code every timer tick the pulse width is increased or decreased by one, so it should go at the same speed in both directions... also, I don't think it's the capacitor, it would have to be pretty big to be able to buffer for seconds given the power that's being drawn.

Regarding PID for the atomizer: you already have kind of a degenerate I-only controller; so not increasing/decreasing by one, but by a number proportional to the voltage difference may be the thing to do. I'll try and play around with that tomorrow.

[cmock]

BTW: I do have a scope; when I mess with the CMR values (last paragraph above), what do I have to look out for to judge whether it's OK or not? I never worked with switching regulators before...

[ReservedField]

WRT VaporWare: their PID tunings didn't work for my coil builds at all, which is to be expected. So I took my PID code, glued it into examples/atomizer, and started playing around with autotuning, which is my main point of interest (and when I get something, I plan on integrating it into VaporWare).

They should work decently across most coils, at least that's our experience. What coil are you using? About autotuning, they found the oscillation point for a P-only loop, applied Ziegler-Nichols to get I and D and then did some manual tuning on top of that (am I correct @s0be?). You could increase P in big steps until it oscillates, then bisect to find the precise oscillation point. I wanted to try it and he suggested I start from P = 27000.

Thanks for the patch. but it seems it doesn't change a thing. I don't understand it, because AFAICT it should now be 10 times quicker, but it doesn't.

There is a point at which feedback loop frequency exceeds the DC/DC converter bandwidth, which means it won't go faster and will over/undershoot. 25kHz may exceed that, but should still be faster than 2.5kHz (considering OFW uses 5kHz). Something's wrong.

What I also don't understand is why decreasing voltage is slower -- from the code every timer tick the pulse width is increased or decreased by one, so it should go at the same speed in both directions... also, I don't think it's the capacitor, it would have to be pretty big to be able to buffer for seconds given the power that's being drawn.

Quick back of the envelope, imprecise calculation. Going from 4V to 2.2V (1.8V drop) means going to 55% of initial voltage. That's about 0.8 time constants. So if it takes 1.5s the time constant should be about 1.9s. Let's say we have a 1ohm coil, it means a 1.9F capacitor. You're right, that's off by orders of magnitude (I know the calculation assumes it's simply discharging, which is not true, but it's way too off to even bother doing it right). The green line looks a lot like an exponential decay, though.

Regarding PID for the atomizer: you already have kind of a degenerate I-only controller; so not increasing/decreasing by one, but by a number proportional to the voltage difference may be the thing to do. I'll try and play around with that tomorrow.

For buck mode (which is were you're working I guessed) I tried changing:

Atomizer_curCmr--;

to:

Atomizer_curCmr -= Atomizer_curCmr > 160 ? 10 : 1;

Without significant results. This introduces a non-linearity by decreasing by 10x of what it increases (but only if it's more than 1/3 of the way, otherwise the converter will runaway to zero resulting in plenty OPENs). I chose 10 because it was close to the 13x I calculated earlier and I didn't need extra checks because of the if(Atomizer_curCmr <= 10) (I'm that lazy).

BTW: I do have a scope; when I mess with the CMR values (last paragraph above), what do I have to look out for to judge whether it's OK or not? I never worked with switching regulators before...

Check that the waveform is clean (no big jumps, no insane ripple). That's all we are after, clean power. And of course look at the slew rate to see if you're making progress. If the waveform is good voltage should be accurate enough, unless you have hardware issues. If the waveform looks good, you should check step response, which is how the converter reacts to a sudden change in output current. We have a purely resistive, fixed load, so we can't really step the output current, but we can abruptly change the output voltage to a much higher or lower value and check how it behaves.

As a side note, check out the pwm144 branch. It's outdated, but if you take the only relevant commit (b470d1d) you shouldn't have problems porting it to the current master, it's very little code. It doubles PWM resolution for free, I have no idea why Joyetech didn't do it. They probably didn't realize PLL runs at twice the CPU clock... Just pointing that branch out in case it's helpful.

Also, feel free to join us at ##evic-modding on Freenode. I'm going to bed now, but I'm usually there all day (CEST time).

[s0be]

The current tunings were not done with finding oscillation and using Z-N calcs. That method is less optimal for thermal systems like this The actual limit to that method is really our output update rate being limited to around 50hz (if I remember correctly, don't have the code handy).

The tunings currently have a bit of a slow rise (intentionally) due to the spurious bad readings we're still trying to figure out. When I have it tuned well, a spurious reading will cause it to quickly ramp voltage up too far and lead to burning.

The method I used: Crank up P until it oscillates, drop it by around 30%, crank up I until it overshoots, lower it. Ignore D. Go back to P and continue to bump it up until the rise time is acceptable, while lowering I. Balance P and I for temperature stability vs rise time. My 'goal' was under 0.5s to temp, with +/-5 deg. F. When I hit that tuning, I found that we still occasionally would get rogue readings (not zero, but definitely not 'real' either), so I lowered P and increased I to dampen the effect rogue readings would have. To work around brown-outs into low ohm coils, I added the 'fire at init watts until 150C' trigger.

@cmock if you're seeing slow voltage rise/fall, check that you've removed/modified the portion that intentionally limits the voltage step rate in your firing loop. Do you have your work posted?

[cmock]

@s0be : OMG that's embarassing. 10 lines above that code I've added some code, and I didn't see it.
@ReservedField : I'm pretty sure that's the reason. But I'll not test that now, because I'm also in CEST and I gotta go to bed.

Regarding tuning (shall we take this to the VaporWare issue section?): I've extensively played with PID tuning on a sous vide thing I built with an atmel uC, and never had success with Z-N, or any method that involved oscillations. What I've done is to find the dead time and the initial gradient and then apply Lambda tuning based on that (at least I think it's lambda, I didn't take very good notes back then), which works very well for heating a few litres of water with 500W off an immersion heater.

Vaping is the same process, although we have no dead time to speak of, so my current approach is to find the time until we reach 150°C with a given power setting, then find the maximum temperature with that power setting, match the exponential curve and derive tau and the gain. From there it's just Lambda tuning again. This should be easy to implement (currently I'm doing it offline), and quick to run on a new coil.

Do you know how the OFW does TC? Because from theory, I don't think there can be a single PID tuning that works for all coil setups -- think about it, one may be a thin gauge single coil with little airflow for a MTL vaper, and the other a heavy-gauge clapton dual-coil for a cloud chaser. Those will have drastically different gains and tau values.

So my idea is to enhance the firmware with a PI autotune function, user sets a wattage they're comfortable with, takes a puff, and magic happens and his TC PI is tuned for this specific setup.

[ReservedField]

Do you know how the OFW does TC?

I can answer this. OFW uses the same method it uses for DC/DC (which I also use), i.e. +/- on power based on the temperature. Kinda like:

if(measured_temp < target_temp) {
    power++;
}
else if(measured_temp > target_temp) {
    power--;
}

[ReservedField]

By the way, I'm experimenting with IIR filters. Spurious reads seem to improve.

[cmock]

IIR filters are a good thing, yup. Another method I've used is to take an odd number of measurements (like 3, 5 or 7) and take the median. This eliminates glitches completely, while with an IIR they do influence the output.

[ReservedField]

IIR filters are a good thing, yup.

Mostly because FIR filters and moving averages need big buffer, and we don't want to waste too much memory.

Another method I've used is to take an odd number of measurements (like 3, 5 or 7) and take the median.

Good call! Here's some testing I did. Note that each filter is duplicated for voltage and current.

First, I set up sampling at 25KHz but feedback at 5KHz, so that I'd have 5 samples per feedback cycle (IIR of course works on the accumulator and not on sample windows).

• First-order lowpass IIR with 5KHz cutoff: works well, but I'd like to lower the cutoff, which means I'd have to use MAC instructions (Cortex-M4 has DSP support), otherwise I loose too much precision to avoid overflow. Not that I'm afraid of asm...
• 5-sample simple average: works very well.
• 5-sample simple median: works well, maybe TC (from VaporWare) is a little less stable? Not perceivable, just looking at the flickering digits.

Then I set up sampling and feedback both at 25KHz (IIR code of course doesn't change as it's not windowed).

• First-order lowpass IIR with 5KHz cutoff: works (as in no OPENs), but VaporWare TC is not really stable, swinging up to -40/+10 °C on a 230°C target.
• 5-sample sliding window average: no OPENs, but TC is -50/+20 on same target.
• 5-sample sliding window median: much better! +/- 15°C.

The result here is that they all work well at low feedback frequency, but the sliding window median shines at higher frequencies. And it makes total sense. In a sense, filters and averages focus on the value, while the median focuses on the trend: the value it provides isn't as accurate, but it's more difficult for spikes or dips to influence it. And we don't need a precise value: in the feedback loop we only look for shorts and opens, while ReadInfo handles the actual resistance measurement. Yes, resistance is calculated in feedback, but I want to remove that and only decide on voltage and current (like I already did for opens). The values are then only checked to less or greater than the target, no precise math.

I'm sure a bigger windowed average would work, but we can't spare that much memory. In ReadInfo we average efficiently because there's no window and we can just sum, while in feedback we can't do that (unless we oversample, like I did in the first series of tests, but then bandwidth is limited). It's easy to see that on a 5-sample window a zero reading would drop the average by 20%, which is substantial.

So I devised a few algorithms to test. The first one works on a sliding window (5 samples in this case). I compute the average of the window. Then I select the samples from the window that are within +/- 20% of the average and I average those, to yield the value used for the control logic. If no samples are within the bounds, I use the whole average. It worked well, but with some caveats. At +/- 10% I had veeeery slow rampup and somewhat less stability, so I raised to +/- 20%. Rampup was still a bit slow, and it had a few moments of instability.

The one I'm vaping on right now seems to work very well. I take the median of the window, then I average it with the average of the window. This smooths out some instabilities, and I seem to be back to +/- 5°C!

[cmock]

I think the general question is: what "speed" do we want from the inner/voltage control loop. The slower it changes the voltage, and the slower it reports back changes (via filtering), the more dead time there will be for the PID to handle.

Dead time limits the speed that the PID can react at, or, vice versa, on a "too speedy" PID setup it leads to oscillations. The flip side of that coin is that more of that dead time allows for better filtering, and with high-gain PID tunings a noisy measurement leads to very wlld variations in the output.

@s0be says he's aiming for 0.5 sec heat up time; someone versed in control theory could probably take that number plus some values for process time constant (~800ms according to my measurements) and dead time (~~30ms) and see whether a stable closed loop is possible and what margin there is for the dead time...

[s0be]

The actual limit we ran into here was having to update the display in the same loop. It takes, about, 7ms to update the screen, so I chose 20ms for the update loop, allowing for more window for the screen to not directly interfere with the pid interval. 10ms on the pid loop would probably work as well (but would need re-tuned).

I agree that someone with more thermal control systems experience than me could probably come up with what a 'best case scenario would be. Ideally, we could just fire at 'X Watts' until we're some arbitrary distance from the target temp then switch over to a pid (we kinda do something like that now with the 150C constant or pidswitch knob). Pids are much easier to tune when they're designed to maintain a steady state rather than find and maintain it.

My previous PID experience was with course correction in robots (IMU applications with kalman filters and whatnot to fuse sensors into a quaternion, then some trig to find what straight actually meant, as they were in motion on non planar surfaces).

[s0be]

Forgot to mention, once we get the threading stuff done and screen display can be moved out, the PID can be massively re-worked to use something like 2 ms update rate (I think 500hz is 'fast enough').

[ReservedField]

Forgot to mention, once we get the threading stuff done and screen display can be moved out, the PID can be massively re-worked to use something like 2 ms update rate (I think 500hz is 'fast enough').

In 20ms (or whatever the preemption quantum is) bursts, though. You need guarantees on timing, why not a timer IRQ? You're not doing enough math for a 500hz ISR to significantly slow down the system.

[s0be]

Fair point, an ISR would probably be the way to go now and later. Would still have to deal with the occasional 7ms delay to update the screen (unless we can safely update the output from inside an ISR, which I do not believe is currently the case), though, which would lead to pretty much the same issue we have now.

[ReservedField]

Setting voltage is just a store, no problem with doing it inside an ISR. Plus your timer ISR and atomizer feedback ISR have the same priority, so they won't preempt each other and mess up the voltage update. IRQs always preempt threads if not masked, so it would be you display thread waiting for PID to finish the iteration and not the other way around.

[s0be]

Perfect, I'll switch over to a timer for the pid and crank it up to 100,000,000Hz. That or maybe my 500Hz I originally thought. If you have the 3/5/7 sample median stuff implemented, I can remove the 'smoothing' of the output voltage ramp up and drop off. Then I'll just need to re-tune the PID this weekend. Can we also safely call Atomizer_ReadInfo from ISR context?

[ReservedField]

Crank it up to 11. About the median stuff, I'll see if I can bring it down from 5 to 3. Right now I just selection sort but I can get more efficient. Need to clean up heavily anyway.
ReadInfo might be an issue. If you're not firing you'll livelock waiting for the ADC IRQ in ADC_Read. If you're firing it'll work from cache, which isn't blocking so no problem (I wrote ADC cache exactly to be able to work on ADC readings from feedback ISR).

[s0be]

Perfect, I can make sure we're firing whenever the timer is running.