Solved – Analysing periodicity in sparse time series

by

I am interested in looking for periodicities in a several day long recording of electrical activity. The traces present a very steady baseline over which, from time to time, some short events (300-500 ms) appear (hence the sparse in the title, although I am not sure it is the right term to use).

Now, the nature of the data prevents me to analyse the trace as a whole (~20 days recorded at 10KHz, my computer will definitely not handle that), so I just wrote a series of routines to find the interesting events, and export their date/time (also their shape, but that is another story) to a text file.

So, I end up with a series of times to analyse.

Just by looking at the raw trace what I can see is:

  1. A ~2 hours periodicity in the events
  2. The events appear in groups. That is, I have a series of 20-30 events in the course of a couple of minutes, then nothing for ~2 hours, then other 20-30 events and so on. Note that the number of events in a group is variable, can be 30 in one and 5 in the next one.
  3. The periodicity is fairly obvious, but it's not perfect: events happen roughly at 2 hours interval, and the exact time may vary from day to day.
  4. There may be superimposed periodicities. In particular, I see events at a specific time of the day, which may or may not fit in the abovementioned 2 hours periodicity.

What can I do to statistically determine the periodicity?

What I have tried to do is to make a frequency histogram of the events, binning every 10 minutes, which empirically seemed like a sensible bin size, and then looking at the ACF or the FFT of the counts, but only in a few cases something pops out with the ACF (nothing with FFT).

So, how would you analyse this type of problem?

Bonus question: in certain cases I have missing data (1 or 2 days missing for technical reasons). How would I account for that?

PS: I am using R, but any non-R solution will do as well!

EDIT

Here is some sample data to play with: http://dl.dropbox.com/u/11676289/exampletimes.txt

Here is a plot of the times:

Plot of the times of the events

and their histogram, in 10 minutes breaks

histogram

Contents hide
1 Best Answer
2 Similar Posts:

Best Answer

Fourier Transform may not be the best tool for analysis of such sparse (and spiked) data. Autocorrelation (perhaps after binning and rescaling) is probably the right tool. Also, one may try using recurrence plot.

However, the process does not look periodic at all. It looks more as two Poission processed – one for releasing spikes, and the second – for the spikes themselves. Take a look at Cox process or materials on spike trains (which exhibits similar behavior the the one from your data).

One can analyze how Fano factor changes with the window size. For all time windows it is $>1$, contrary to what one could expect for a periodic process (for a periodic process one should get $<1$ for time window related to its periodicity); for Poisson process it should be just $1$.

enter image description here

Also, you may try to look at the following:

Similar Posts:

Rate this post

Leave a Comment

Solved – Analysing periodicity in sparse time series

by

I am interested in looking for periodicities in a several day long recording of electrical activity. The traces present a very steady baseline over which, from time to time, some short events (300-500 ms) appear (hence the sparse in the title, although I am not sure it is the right term to use).

Now, the nature of the data prevents me to analyse the trace as a whole (~20 days recorded at 10KHz, my computer will definitely not handle that), so I just wrote a series of routines to find the interesting events, and export their date/time (also their shape, but that is another story) to a text file.

So, I end up with a series of times to analyse.

Just by looking at the raw trace what I can see is:

  1. A ~2 hours periodicity in the events
  2. The events appear in groups. That is, I have a series of 20-30 events in the course of a couple of minutes, then nothing for ~2 hours, then other 20-30 events and so on. Note that the number of events in a group is variable, can be 30 in one and 5 in the next one.
  3. The periodicity is fairly obvious, but it's not perfect: events happen roughly at 2 hours interval, and the exact time may vary from day to day.
  4. There may be superimposed periodicities. In particular, I see events at a specific time of the day, which may or may not fit in the abovementioned 2 hours periodicity.

What can I do to statistically determine the periodicity?

What I have tried to do is to make a frequency histogram of the events, binning every 10 minutes, which empirically seemed like a sensible bin size, and then looking at the ACF or the FFT of the counts, but only in a few cases something pops out with the ACF (nothing with FFT).

So, how would you analyse this type of problem?

Bonus question: in certain cases I have missing data (1 or 2 days missing for technical reasons). How would I account for that?

PS: I am using R, but any non-R solution will do as well!

EDIT

Here is some sample data to play with: http://dl.dropbox.com/u/11676289/exampletimes.txt

Here is a plot of the times:

Plot of the times of the events

and their histogram, in 10 minutes breaks

histogram

Best Answer

Fourier Transform may not be the best tool for analysis of such sparse (and spiked) data. Autocorrelation (perhaps after binning and rescaling) is probably the right tool. Also, one may try using recurrence plot.

However, the process does not look periodic at all. It looks more as two Poission processed – one for releasing spikes, and the second – for the spikes themselves. Take a look at Cox process or materials on spike trains (which exhibits similar behavior the the one from your data).

One can analyze how Fano factor changes with the window size. For all time windows it is $>1$, contrary to what one could expect for a periodic process (for a periodic process one should get $<1$ for time window related to its periodicity); for Poisson process it should be just $1$.

enter image description here

Also, you may try to look at the following:

Similar Posts:

Rate this post

Leave a Comment