PayPal IPN Failure: 500 Internal Server Error - Read #610

protectivediet · 2015-07-02T05:57:22Z

All my IPN information hasn't been adjusted in months. It is up to date and correct however s2member stopped recognizing notifications sent from paypal sometime after midnight and 5:05 AM Pacific Time on July 1, 2015.

codeforest · 2015-07-02T08:59:25Z

Confirming the same issue

jaswrks · 2015-07-02T13:11:39Z

Confirmed. First reported here: https://wordpress.org/support/topic/error-unable-to-verify-_post-vars-since-some-updates-nginx?replies=4#post-7135481

Taking a look now. One moment please.

jaswrks · 2015-07-02T13:19:50Z

We are currently investigating this issue and I will post additional details as we learn more. So far this appears to be a connectivity issue with PayPal's IPN service; i.e., PayPal servers are experiencing problems.

In at least two cases that I've investigated so far, PayPal's IPN server was returning a 500 Internal Server error whenever verification was attempted by s2Member. This indicates they are having problems on the PayPal side.

We also have tickets in with PayPal MTS on this matter.

NOTE: You can confirm the server connectivity issue by enabling s2Member's logging routines and then looking inside s2-http-api-debug.log file for connection failures.

[url] => https://www.paypal.com/cgi-bin/webscr
    [response] => Array
        (
            [headers] => Array
                (
                    [server] => AkamaiGHost
                    [mime-version] => 1.0
                    [content-type] => text/html
                    [content-length] => 283
                    [expires] => Wed, 01 Jul 2015 21:20:44 GMT
                    [date] => Wed, 01 Jul 2015 21:20:44 GMT
                    [connection] => close
                    [strict-transport-security] => max-age=63072000
                )

            [body] => <HTML><HEAD>
<TITLE>Internal Server Error</TITLE>
</HEAD><BODY>
<H1>Internal Server Error - Read</H1>
The server encountered an internal error or misconfiguration and was unable to
complete your request.<P>
Reference #3.4035d417.1435785644.f7df9a3
</BODY></HTML>

            [response] => Array
                (
                    [code] => 500
                    [message] => Internal Server Error
                )

            [cookies] => Array
                (
                )

            [filename] =>
        )

)

jaswrks · 2015-07-02T13:20:10Z

UPDATE: We believe that we have a solution. I will post details momentarily.

…See: #610

jaswrks · 2015-07-02T13:46:51Z

Description of Problem

PayPal's IPN verifier started kicking back 500 Internal Server Error yesterday; i.e., there seems to be some trouble on the PayPal side. This impacts the live PayPal servers only, as it does not occur in the sandbox system. Live is what matters though!

We have tickets open with PayPal MTS in an effort to resolve this quickly.

What is the IPN Verifier?

PayPal's IPN verifier is a critical component for any site running PayPal integration, and in particular, any site that is using PayPal Standard Buttons instead of s2Member Pro-Forms. Without this key component, s2Member transactions will fail post-processing in many cases. In other words, the transaction goes through, but you will need to setup the customers's account manually due to PayPal's inability to verify the IPN data that s2Member receives.

For example, you would need to:

Log into your WP Dashboard
Change the customer's Role to an s2Member Level #.
Add the transaction details yourself, and list any Custom Capabilities they should have based on the purchase that they made.
Email the customer with an apology and their account details.

Why is PayPal Returning 500 Error Codes?

We dont' have confirmation yet. However, we have found it to be related to the default User-Agent header that is sent by the WordPress WP_Http class. The default User-Agent is WordPress/[version]; [your site URL].

PayPal (as of yesterday) seems to refuse that sort of connection. We suspect this is an error on their part, but we are awaiting their confirmation on the matter.

What Is s2Member Doing About This?

This is a PayPal issue. We are contacting PayPal MTS and hoping for them to implement a fix as quickly as possible. I will post an update just as soon as I hear back from their tech. support dept.

That said, we have run some preliminary tests, and it appears that we can work around this problem by setting a User-Agent header that is plugin-specific; i.e., avoid the problematic User-Agent header that PayPal seems to refuse as of yesterday.

I will post a short-term workaround momentarily and make it available for testing/feedback.

jaswrks · 2015-07-02T13:50:15Z

Short-Term Workaround (Possible Fix)

Please download this updated file and upload it via FTP. Allowing it override your existing copy of s2member/includes/classes/utils-urls.inc.php
Run a test transaction and confirm that the problem no longer impacts you.
Report back with your findings, so that we can gather feedback and get a better sense of how this might work across a large number of sites. The fix seems to resolve the issue in our lab.

evrelia · 2015-07-02T14:00:56Z

Just to confirm - I am experiencing the exact same issue. However, I only appear to have issues with cancellations (I.e, user cancels their recurring billing, PayPal sends a cancellation IPN causing s2 to set an EOT time). Normal sales are all going through fine for me, using pro-forms.

Also interestingly, if you view the IPN history on PayPal, almost all of the message IDs from the last day lead to "service unavailable - zero size objects" on the PayPal site.

Can I confirm the above fix is safe to implement in my current situation? My site is very active with around 150 sales each day. I wouldn't want to risk causing the sales to stop working just to fix the cancellations.

jaswrks · 2015-07-02T14:17:59Z

@evrelia writes...

Can I confirm the above fix is safe to implement in my current situation? My site is very active with around 150 sales each day. I wouldn't want to risk causing the sales to stop working just to fix the cancellations.

If you're running Pro-Forms, the impact is minimal, as you mentioned. I don't see any problem occurring if you implement the workaround I posted above. However, given the minimal impact that you're seeing, if I were you I'd await additional feedback from others in this thread. We should see others respond here shortly and that might help you decide.

jaswrks · 2015-07-02T14:50:53Z

Confirmation here that the fix posted above works for at least one other plugin developer as well.

jaswrks · 2015-07-02T14:59:31Z

A customer writes...

I’m afraid I don’t have an account on GitHub to leave feedback right at this moment, but I wanted to let you know ASAP that YES, an initial test shows that the fix of tweaking the user-agent string successfully enabled one of my sites to process an IPN which I re-sent manually from the PayPal interface.

Thank you thank you thank you thank you!!!

kkranzo · 2015-07-02T15:18:36Z

Using a different plugin and was wondering what should be changed to make this work in the below code?

$request = new WP_Http();
$response = $request->post($paypal_url, array('ssl' => true, 'body' => $req));

msykes · 2015-07-02T15:20:02Z

Same here, I can confirm the fix solves the problem for me. Changing the user agent also fixes the issue for Events Manager Pro so I must thank you for both fixing my s2member installation and also providing a solution to my plugin users. One more reason why I recommend s2member as a great companion to EM!

msykes · 2015-07-02T15:22:13Z

@kkranzo

$response = $request->post($paypal_url, array('ssl' => true, 'body' => $req, 'user-agent'=>'plugin name for example'));

jaswrks · 2015-07-02T15:39:26Z

A customer writes...

Yes - this seems to have worked!
At least it did on my test transaction - with no more error.

jaswrks · 2015-07-02T15:42:40Z

@msykes writes...

Same here, I can confirm the fix solves the problem for me.

Glad to hear it. Thanks for your kind words. Back at ya :-)

codeforest · 2015-07-02T16:07:48Z

@jaswsinc Works for me too...

andrejstas · 2015-07-02T16:14:04Z

Great job! Works for me too!

codeforest · 2015-07-02T16:45:26Z

I am the Lead Developer of OptimizePress, we have a patch ready for OptimizeMember, please contact our support team. Or, download a patch here: https://www.dropbox.com/s/nklxunlj289gylc/OPM_PayPal_patch.zip?dl=0 Just unzip it in your OptimizeMember plugin folder, the path is in the archive already...

jaswrks · 2015-07-02T17:01:25Z

@codeforest writes...

Works for me too... so, something changed on PayPal's side that was causing this?

Glad to hear it :-) Yeah, I'm not entirely clear on the specifics on their side (yet), but for whatever reason they started failing on a default WordPress User-Agent header.

Dean Robinson (an Event Expresso author) writes...

Over at Event Espresso we have had a similar issue - it's definitely from PayPals end.

As Jason Caldwell stated, its the User Agent issue. The devs here have advised that "its due to paypal expecting unique user-agents on ipn validation per customer account. So they need to make sure that when they do the ipn validation they include a user-agent string in the request that is unique to that customer account."

nerrad · 2015-07-02T18:41:59Z

Have you heard anything from paypal yet? Here at Event Espresso, we were testing the user-agent fix on our servers, but then we've started seeing things work again without the fix. So it appears that the paypal minions woke up and saw the red flashing light. Unfortunately doesn't seem to be any news from paypal about what was going on...

josiahmann · 2015-07-02T19:22:01Z

This temporary fix appears to be working. I've had three registrations that have worked. Thank you.

jaswrks · 2015-07-02T19:28:58Z

@nerrad writes...

Have you heard anything from paypal yet?

Only that they are aware of the issue and that they are working hard to resolve it quickly. I haven't gotten anything that confirms it has been fixed yet though.

@nerrad writes...

So it appears that the paypal minions woke up and saw the red flashing light.

I can confirm this also. I'm not seeing any other reports of this in the past hour or so, and my tests show that the issue has been resolved on PayPal's side now. Let's hope it stays that way.

👏 Hooray for WP plugin developers teaming up this! I really appreciate everyone's help in not only reporting this to us, but also helping us flood PayPal with reports of this. That's what gets their attention.

msykes · 2015-07-02T21:25:00Z

´flood PayPal with reports of this. That's what gets their attention.´
Maybe we can tweet/FB them start a hashtag, open to suggestions ;) #‎PayPalLetWPIn‬

jaswrks · 2015-07-03T13:59:19Z

@msykes writes...

Maybe we can tweet/FB them start a hashtag, open to suggestions ;) #‎PayPalLetWPIn

I'm open to that, happy to join you :-) From past experience (maybe others can share their own), the only way I can actually get connected to anyone in their tech. dept. is via PayPal MTS; and even this is very slow.

So yeah, maybe we could all hit them on Twitter next time. The hashtag #PayPalLetWPIn seems like a good one to me. Mention: @PayPal_MTS

jaswrks · 2015-07-03T14:02:00Z

PayPal wrote me back late yesterday and asked if I could confirm the issue had been fixed. I told them that my own tests indicated that it had been, and I asked them to confirm that the issue has been fixed on their end. They haven't written me back yet.

So while that's not a confirmation from them that the issue has in fact been resolved entirely; my feeling is that is has, and they are gathering feedback now.

msykes · 2015-07-03T14:33:02Z

same here, got a similar message and it seems to work now... I guess no need for a crusade :-)

ghost · 2015-07-05T11:59:27Z

2 questions:

Does anyone know exactly when the problem started? We were in the middle of a membership drive and had all sorts of failures on June 30 and into July 1. I've laboriously gone back as far as June 27 cross-footing PayPal activities versus membership and ~~think~~ I've corrected everything. (grrrrrrr!) Need I look further back?
Jason: what's your confidence level that PayPal has reverted or fixed the cause ... or, in other words, shall we continue with your fix to utils-urls.inc.php ?

THANKS Jason, and others, for so quickly jumping on PayPal to get it resolved.

jaswrks · 2015-07-07T12:34:08Z

Does anyone know exactly when the problem started?

Difficult to say. From what I can tell, PayPal started to experience this problem in small spurts as far back as June 30. That's the earliest report that I've seen.

Jason: what's your confidence level that PayPal has reverted or fixed the cause ... or, in other words, shall we continue with your fix to utils-urls.inc.php

All indications are that the problem has been solved; i.e., we are not seeing this reported any longer, and my own tests indicate that it has been resolved. Thus, the short-term fix posted above is no longer necessary. I am going to close this issue. If anyone experiences this again, please feel free to leave comments and we can reopen if necessary.

Note: If you have already implemented the short-term fix, you can remove it. That said, there is really no reason to remove it; because it simply alters the User-Agent header in a way that better identifies s2Member. The change introduced by the short-term fix is actually going to be included in the next official release of s2Member, since we consider that to be a slight improvement.

jaswrks added bug high priority labels Jul 2, 2015

jaswrks changed the title ~~No longer communicating with paypal~~ PayPal IPN Failure: 500 Internal Server Error - Read Jul 2, 2015

jaswrks self-assigned this Jul 2, 2015

jaswrks pushed a commit that referenced this issue Jul 2, 2015

PayPal workaround. Use non-default User-Agent provided by WordPress. …

a33aab0

…See: #610

jaswrks pushed a commit that referenced this issue Jul 2, 2015

PayPal workaround. Use non-default User-Agent provided by WordPress. …

178e1fc

…See: #610

jaswrks closed this as completed Jul 7, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PayPal IPN Failure: 500 Internal Server Error - Read #610

PayPal IPN Failure: 500 Internal Server Error - Read #610

protectivediet commented Jul 2, 2015

codeforest commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

evrelia commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

kkranzo commented Jul 2, 2015

msykes commented Jul 2, 2015

msykes commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

codeforest commented Jul 2, 2015

andrejstas commented Jul 2, 2015

codeforest commented Jul 2, 2015

jaswrks commented Jul 2, 2015

nerrad commented Jul 2, 2015

josiahmann commented Jul 2, 2015

jaswrks commented Jul 2, 2015

msykes commented Jul 2, 2015

jaswrks commented Jul 3, 2015

jaswrks commented Jul 3, 2015

msykes commented Jul 3, 2015

ghost commented Jul 5, 2015

jaswrks commented Jul 7, 2015

PayPal IPN Failure: 500 Internal Server Error - Read #610

PayPal IPN Failure: 500 Internal Server Error - Read #610

Comments

protectivediet commented Jul 2, 2015

codeforest commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

Description of Problem

What is the IPN Verifier?

Why is PayPal Returning 500 Error Codes?

What Is s2Member Doing About This?

jaswrks commented Jul 2, 2015

Short-Term Workaround (Possible Fix)

evrelia commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

kkranzo commented Jul 2, 2015

msykes commented Jul 2, 2015

msykes commented Jul 2, 2015

jaswrks commented Jul 2, 2015

jaswrks commented Jul 2, 2015

codeforest commented Jul 2, 2015

andrejstas commented Jul 2, 2015

codeforest commented Jul 2, 2015

jaswrks commented Jul 2, 2015

nerrad commented Jul 2, 2015

josiahmann commented Jul 2, 2015

jaswrks commented Jul 2, 2015

msykes commented Jul 2, 2015

jaswrks commented Jul 3, 2015

jaswrks commented Jul 3, 2015

msykes commented Jul 3, 2015

ghost commented Jul 5, 2015

jaswrks commented Jul 7, 2015