Suboptimal performance in the presence of noisy data

[young-oct]

Hi there,

I want to start off by thanking you for providing this implementation which I found very helpful in doing my own research work.
I created a modified version of the spsa based on your implementation.

I noticed that the performance of spsa seems to be negatively affected by the amount of noise in the dataset, as the gradient seems to be trapped in some local minimum, resulting in the cost function's plateau. Please see the comparison from the attached diagrams.

I wonder if you could offer some advice on handling noisy data.

[rajcscw]

Hi there! What is the size of the parameter that you are optimizing with SPSA? If it is too large, you can try to approximate the gradients by having more pairs of perturbations and averaging them ( as done in https://www.researchgate.net/publication/335698996_Leveraging_Domain_Knowledge_for_Reinforcement_Learning_Using_MMC_Architectures) ...May be this helps with your noisy data problem too.

[young-oct]

@rajcscw Thank you for getting back to me so quickly. Here is how I set up the dummy problem to better understand what SPSA does before working on the real application. I also wonder if Adam can help?

N = 500
input_dim = 10
output_dim = 1

np.random.seed(13)
x = np.random.rand(input_dim, N)
W_true = np.random.rand(output_dim, input_dim)
print("The true value of W is: \n " + str(W_true))
noise = np.random.rand(N)
y = np.matmul(W_true, x) + noise * 2

[rajcscw]

Couple of things I noticed:

• May be the magnitude of the noise is too high? It is almost in the same magnitude as the data?
• Wondering if you should also learn the bias term y = np.matmul(W_true, x) + W_bias * noise * 2

[young-oct]

@rajcscw That's a good point. I'm trying to replicate what this paper has demonstrated using SPSA.

https://opg.optica.org/oe/fulltext.cfm?uri=oe-28-16-23306&id=433716 in my repo

and I observed a similar behaviour in that the cost function that it plateaus early and seems to not descend in the desired direction.