Exclude frequency range of line noise when fitting slope

[srcole]

I'm fitting a PSD (see below) for a signal after I removed line noise (data available in data.mat here). This biases the slope fit down. It might be a good idea to have a kwarg that allows the user to define frequency range(s) to ignore during the fit.

[bringingjoy]

Sub-range(s) to ignore during full model fitting are now available under the dev branch. Given the potential plural, the optional parameter expects a list of lists (and currently also works with lists of tuples).

[bringingjoy]

Dev branch is updated with the ignore_range procedure

[TomDonoghue]

Let's not close issues until they are merged to master.

This looks great, thanks @bringingjoy !

From playing with it a little:

• Right now it interpolates in linear space, which makes it curved in log space, which is perhaps not ideal since the assumption is generally that the PSD is linear in log space.
• Perhaps problematic that it uses interpolated points in slope fitting, and can fit oscillations over 'ignored' regions.

^These things might be moot (unlikely / aren't really relevant) in the use case of excluding small regions, such as line noise, but creep in more if people try to use it to exclude broader ranges, and we should be aware of implications of various ways people might use FOOOF.

I know I suggested this approach, these open issues are totally my bad.

Let me think about it a bit / play around, and see where to go from here.

[bringingjoy]

I'm sorry about closing the issue preemptively. Don't apologize - I had proposed the linear interpolation, but certainly didn't exhaust testing corner cases, so sorry for the addition of any such potential problems! It's possible that the replaced region's use during slope fitting could negatively impact the resulting fit, but cannot think of a better alternative for this presently - in many use cases, at least, it improves what would otherwise be a poor fit.

As far as fitting "oscillations" introduced by this method, we could exclude oscillations detected on the ignored interval +/- some factor of its bandwidth, or a similar constraint from being added to the overall curve.

Had another idea for accommodating slope fitting with ignoring regions, but could become much less feasible with many ranges specified, so will keep you posted if I think of anything else.

[rdgao]

I am demanding for this to happen for v1.0!

[parenthetical-e]

Rather than try and solve all exclusion cases, limit the problem? Add a way to handle only line noise, and so assume/enforce a single exclusion of ‘narrow' range. With such a constraint, masking or interpolation should be viable? This also greatly limits what you need to think though, reason about, etc, so on, to merge this feature. (e)

On Sep 22, 2017, at 12:08 AM, bringingjoy ***@***.***> wrote: I'm sorry about closing the issue preemptively. Don't apologize - I had proposed the linear interpolation, but certainly didn't exhaust testing corner cases, so sorry for the addition of any such potential problems! It's possible that the replaced region's use during slope fitting could negatively impact the resulting fit, but cannot think of a better alternative for this presently - in many use cases, at least, it improves what would otherwise be a poor fit. As far as fitting "oscillations" introduced by this method, we could exclude oscillations detected on the ignored interval +/- some factor of its bandwidth, or a similar constraint from being added to the overall curve. Had another idea for accommodating slope fitting with ignoring regions, but could become much less feasible with many ranges specified, so will keep you posted if I think of anything else. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#20 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABIxh5m6FVgE3hns61zwvhRj0sP-k2cWks5sk1z_gaJpZM4O_Vxt>.

(e)

[TomDonoghue]

Bumping to add after v1.

[TomDonoghue]

Okay - so I (finally) add a simple solution to a restricted problem space for excluding regions. To limit the scope and avoid some of the problems I think would arise from dealing with large / multiple frequency ranges, and/or dealing with discontinuities, the minimal solution I've added is to interpolate narrow frequency regions in the spectrum itself, prior to fitting. These regions are interpolated as a linear interpolation in log-log space, under the 1/f assumption. This is implemented as a new function interpolate_spectrum, which is now in the utils sub-module. This is done outside the model object, so that it's up to the user to apply this function before model fitting.

I think this solves the general case of line noise (or line noise related bandstop filtered regions), and so deals with the immediate use case. A broader discussion of fitting different ranges or excluding regions is available here: fooof-tools/Development#8

[TomDonoghue]

Closing this now - future discussion can move to fooof-tools/Development#8, unless anyone wants to re-open with any issues with interpolate_spectrum.