How to find the noise floor?
I have a data source that at times produces very clean output and at other times produces noisier output. The signal proper comes in bursts, and its amplitude when the noise is low is sometimes lower than the noise floor when the output is noisy. In other words, the output looks like this:
The little peaks on the left are also real signal and I'd like to detect them, so I need the software to be able to set an appropriate amplitude threshold at all times. My problem is that every method I've come up with (e.g. measuring the standard deviation for the last minute) is thrown off by the signal peaks, so if there are two peaks somewhat close together I might miss the second one. Either that or the algorithm starts detecting noise as signal.
How does software normally handle this? I've never done much DSP so I have no idea.
|
|
How does software normally handle this? I've never done much DSP so I have no idea.
one way is to put in 2 sensors and do what you were doing... a time window, across both sensors, weighted average.
another attack would be to, if possible, find and eliminate the source noise or reduce it. Is the noise periodic? Is it tied to another device powering up? can you relocate the device? Etc?
you can also try number tricks. A FFT could help, depends on what the data looks like.
pattern recognition can help, if there is some pattern to the data you expect to either the noise or the real data? Either direction could help take a 'better guess' as to valid or noise.
any more info might help... ?
another attack would be to, if possible, find and eliminate the source noise or reduce it. Is the noise periodic? Is it tied to another device powering up? can you relocate the device? Etc?
you can also try number tricks. A FFT could help, depends on what the data looks like.
pattern recognition can help, if there is some pattern to the data you expect to either the noise or the real data? Either direction could help take a 'better guess' as to valid or noise.
any more info might help... ?
Last edited on
More information: the data is the similarity score for the consecutive frames captured by a camera. I want to detect motion while reducing useless footage. The noise is caused by frames captured in darkness being more dissimilar to each other due to the gain being raised so much the noise from the CCD becomes obvious.
Mmh... Right now the data is score(frame[i], frame[i - 1]). The analog of a second sensor would be, for example, score(frame[i], frame[i - 2]). That's easily doable, but how would I combine both sequences?
| one way is to put in 2 sensors and do what you were doing... a time window, across both sensors, weighted average. |
| The noise is caused by frames captured in darkness |
Can you adjust the threshold according to the darkness level?
you would not combine the 'frames' you combine the 'data'. That is, for a given moment in time, do both frames have motion, or only one frame? If its one frame, and its also dark, odds are its noise?
going with Dutch, adjusting something based off the light level is a good idea if possible.
on the 2 sensor idea, instead of double down, maybe get an IR camera instead, and weight it to trust it more in the dark and weight it to trust the normal one more when there is good lighting? But I don't know if your design allows for more costs/power draw/ space/ etc... ?
since it is imagery, you can also consider doing a quick scan of the 'motion'. How big are the pixels moving around? Are they bands/lines across the image that do not match 'normal, worth capturing' motion? Are they pixels all over the place, also not matching 'normal'? That would let you add a little processing to make a smarter decision. you can sometimes do work like this on a lower resolution copy of the image rather than pixel by pixel a 2000x2000+ sized monster. This has a double effect of removing most of the pixel type noise and keeping only larger things, and its faster. ?? But I am just throwing stuff around ...
going with Dutch, adjusting something based off the light level is a good idea if possible.
on the 2 sensor idea, instead of double down, maybe get an IR camera instead, and weight it to trust it more in the dark and weight it to trust the normal one more when there is good lighting? But I don't know if your design allows for more costs/power draw/ space/ etc... ?
since it is imagery, you can also consider doing a quick scan of the 'motion'. How big are the pixels moving around? Are they bands/lines across the image that do not match 'normal, worth capturing' motion? Are they pixels all over the place, also not matching 'normal'? That would let you add a little processing to make a smarter decision. you can sometimes do work like this on a lower resolution copy of the image rather than pixel by pixel a 2000x2000+ sized monster. This has a double effect of removing most of the pixel type noise and keeping only larger things, and its faster. ?? But I am just throwing stuff around ...
Last edited on
| Can you adjust the threshold according to the darkness level? |
| maybe get an IR camera instead |
| since it is imagery, you can also consider doing a quick scan of the 'motion'. How big are the pixels moving around? Are they bands/lines across the image that do not match 'normal, worth capturing' motion? Are they pixels all over the place, also not matching 'normal'? |
I don't think that is outside DSP capability.
motion is basically which pixels changed more than some amount since last frame, is that how you are doing it?
if so, I believe a dsp can handle drawing (conceptually; really just tracking a few numbers no actual drawing) like a bounding rectangle around the changed pixels and getting the area. If that is bigger than X (10 for a 3x3 or bigger block, 16 for a 4x4 or bigger, etc) or you can do it by dimension (if height >2 and width > 2) or the like?
Or are you detecting the motion another way? Or am I missing something? That could be too slow, but the chip can do the math... not up on current dsp speeds?
motion is basically which pixels changed more than some amount since last frame, is that how you are doing it?
if so, I believe a dsp can handle drawing (conceptually; really just tracking a few numbers no actual drawing) like a bounding rectangle around the changed pixels and getting the area. If that is bigger than X (10 for a 3x3 or bigger block, 16 for a 4x4 or bigger, etc) or you can do it by dimension (if height >2 and width > 2) or the like?
Or are you detecting the motion another way? Or am I missing something? That could be too slow, but the chip can do the math... not up on current dsp speeds?
Last edited on
Right now I'm using the pHash algorithm, which for an image gives me back a string of bits where the Hamming distance to another string of bits is the dissimilarity between the two images, more or less.
This is running on a normal desktop CPU, but I have a strict time limit per image, otherwise the process will take an unreasonable or even infinite amount of time. Hence all I have time for is very basic processing.
This is running on a normal desktop CPU, but I have a strict time limit per image, otherwise the process will take an unreasonable or even infinite amount of time. Hence all I have time for is very basic processing.
I can process multiple images as a single work unit, but I can only trade so much time for accuracy.
Are you able to predict the output either mathematically or functionally?
If so, would it not be able to implement principals of process control whereby, if actual falls outside of tolerance range of predicted output then it gets rated as noise.
If so, would it not be able to implement principals of process control whereby, if actual falls outside of tolerance range of predicted output then it gets rated as noise.
If I have no other choice I could use the time of day to set the threshold, but I wanted to avoid that. I feel like this should be possible using only the image content.
It would be ... but you are not using the content, you are using a hash of the content. The only ways I know of to do this look at it pixel wise... that algorithm tells you that two images are close, but it can't tell you if it has noise vs actual data changes in any way I was able to find digging around earlier. I don't see an easy modification to that algorithm to do anything here either.
Last edited on
Ah, screw it. I'll just use a multiple of the moving standard deviation as the threshold. Thanks for the help, guys.
Topic archived. No new replies allowed.