# Fun with speckle patterns

I recently came across a really fascinating video – https://www.youtube.com/watch?v=tYFLze9VwB0 where they make use of a micro laser projector to generate a speckle pattern on a surface and then photograph it with a DSLR. They then simply touch a surface with their hand/finger and then take a photo of the speckle pattern again after the hand has been removed from the surface and compare the two speckle patterns to see the areas that have changed.

See their paper here – https://ieeexplore.ieee.org/document/6247655. They note the “approach takes advantage of the fact that tiny surface deformations cause phase changes in reflected coherent light which alter the speckle pattern visible under laser illumination”.

I was curious if I could replicate their experiment using a laser pointer and a microscope objective (25x) as a beam expander, to generate a larger beam diameter.

In order to capture images from my DSLR I made use of gphoto2 and tethered the camera to my computer, so that I could avoid having to manually press the shutter button, which I found would cause vibrations and alter the captured speckle pattern.

```gphoto2 --capture-image-and-download
```

The following image shows the laser pointer pushed slightly inside a 25x microscope objective:

This image depicts a speckle pattern on a wall, simply using a laser pointer and pointing it through a 25x microscope objective:

I then touched the surface with my hand and removed it, and took another photograph of the speckle pattern.

You can hopefully make out my hand in the following image, created by finding the differences between two speckle patterns:

The following simple code was used to extract the differences between the two speckle patterns, I picked the red channel from the image and found out about this way to ‘difference’ two images from – https://www.pyimagesearch.com/2017/06/19/image-difference-with-opencv-and-python/

```import cv2
import numpy as np
from skimage.metrics import structural_similarity as compare_ssim

(score, diff) = compare_ssim(a, b, full=True)
diff = (diff * 255).astype("uint8")
thresh = cv2.threshold(diff, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]

cv2.imwrite("out.jpg", thresh)
```

Alternatively if you image the ‘diff’ data directly, without a threshold, the image looks like:

I’m just thinking about ways I could remove some of the noise, I’m currently trying different sizes of a kernel along with different numbers of iterations.

```kernel = np.ones((3,3), np.uint8)
thresh = cv2.erode(thresh, kernel, iterations=3)
```

Gives an image like below:

## Extracting keypresses from a calculator

The following image shows the calculator attached to a vice, to maintain stability, which is very important.

This image shows the calculator in the same position, with the keys 7856 pressed.

Using the same code snippet as above, you should be able to see keys 7856 where pressed from the difference of the two speckle patterns (without looking at the display ;).

Alternatively if you image the ‘diff’ data directly, without a threshold, the image looks like:

## Speckle pattern difference changing over time

I took a photo of an initial speckle pattern then touched my hand on the wall and continued taking photos every 10s, creating a difference between the new photo and the initial speckle pattern each time.

I think temperature seems to play some part in the decay potentially. I noticed in a warm room I was finding it hard to pick up an image of my hand, when I opened the door and let it cool a bit, it seemed to give me a better image. This needs more experimentation!

Without threshold:

## Code

https://github.com/anfractuosity/specklepattern – the code for using gphoto to take photos and also generating the above video

## To Do

Initial experiments were conducted with a Tamron 17-50mm lens on a Canon 550D, it would be fun to try a more telephoto lens such as a 70-200mm lens I’ve got.

Also I would like to try a 100x microscope objective and a greater difference of the laser to surface. This might require a higher power laser though.

## Related work

Also I found there’s interesting research using thermal cameras to obtain keypresses:

Heat of the Moment: Characterizing the Efficacy of Thermal Camera-Based Attacks – https://www.usenix.org/legacy/events/woot11/tech/final_files/Mowery.pdf

• ##### Jorge Stoilov

Hi!,
Impressive and very interesting! If I found my laser pointer I’ll try. Related to reduce noise, did you try a median stack? I mean taking for example 5 shots before, make a median stack with them and another 5 after and make another median stack before comparing them.
Example:https://patdavid.net/2013/05/noise-removal-in-photos-with-median_6

Hi, I’ve not tried that actually, but it’s a good idea. I’m not sure though if the noise might be something to do with the speckle pattern changing though, rather than noise from the camera sensor.

• ##### DaveE

I would think that the speckle pattern would be static, other than the slow “fade” that you observed, in which case a temporal median function wouldn’t help any.

OTOH, reaching (way) back to my days in the aerospace industry, synthetic-array radar (SAR) images had strong speckle, and 2D median filters worked very well at getting rid of it, with surprisingly good retention of the actual subject detail (at least relative to the size of the median kernel). The resulting images were somewhat blurred, but were *much* easier to distinguish subjects in.

OpenCV has a median function in it, you should try playing with it and see if it helps. The optimal kernel size will depend on how many pixels the noise structures spread over. IIRC, just 3×3 or 5×5 kernels worked well for our SAR images, but you might need to go quite a bit higher, as I suspect the speckle artifacts here are large relative to the sensor pixels.

*Fascinating* project(!!), good luck with your further exploration!