Bart,
the point about doing filtering as late a possible is it’s good practice for the reason stated. Since R-M is a straight sum operation it likely does not matter if you do it before or after in this case.
The problem is that of using running mean as a filter. It is that which produces the variations in the plot that change shape as you change the filter period. Either you get a wrong result and differentiate or you differentiate and get the wrong result later. As you demonstrate it’s the same wrong result’.
If you want to try a filter that does not let large lumps stop band frequencies get past it you could try the following awk script.
#!/bin/awk -f
# pass input through 3 sigma gaussian filter where sigma, if not given, is 2 data points wide
# usage : ./gauss.awk filename
# optional scale_factor simply scales the output
# sigma can be compared to the period of the -3dB point of the filter
# result is centred, ie not shift. dataset shortened by half window each end
# data must be continuous and equally spaced
# jan2011 , up to 8.6f for month precision consistency and better FFT
BEGIN{ OFMT = “%8.6f”
# ARGV[1]=filename; argv[0] is script name, hence ARGC>=1
pi= 3.14159265359811668006
if ( ARGC >3 ) {scaleby=ARGV[3];ARGV[3]=”"} else {scaleby=1};
if ( ARGC >2 ) {sigma=ARGV[2];ARGV[2]=”"} else {sigma=2};
print “filtering “ARGV[1]” with gaussian of sigma= “,sigma
root2pi_sigma=sqrt(2*pi)*sigma;
two_sig_sqr=2.0*sigma*sigma;
gw=3*sigma-1; # gauss is approx zero at 3 sigma, use 3 sig window
# eg. window=2*gw-1 – 5 pts for sigma=1; 11pts for sig=2; 3 sig=17
# calculate normalised gaussian coeffs
for (tot_wt=j=0;j<=gw;j++) {tot_wt+=gwt[-j]=gwt[j]=exp(-j*j/two_sig_sqr)/ root2pi_sigma};
tot_wt=2*tot_wt-gwt[0];
tot_wt/=scaleby;
for (j=-gw;jgsfile;
ln=-1;
}
{
xdata[++ln]=$1;
ydata[ln]=$2;
if (ln>2*gw)
{
gauss=0
for (j=-2*gw;j> gsfile;
}
else
{
# print $1,$2;
}
}
END {
print “#gaussian window width = “gw+gw+1″,done”
print “#output file = “gsfile
}
[/sourcecode]