Fatal Failure / 400 Bad Request #117

rakiru · 2015-11-05T23:40:35Z

Some seemingly random small percentage of IPNs fail with

Invalid postback. (<html> <body> Fatal Failure <br> </body> </html> )

We can't find any way to reliably reproduce the error, resorting to manually making payments repeatedly until it triggers. A Google search of "Fatal Failure" gives a few results, but no real suggestions on the cause, so I was wondering if you had seen this before.

Sometimes PayPal will attempt the IPN a few more times, but it will either error in the same way, or django-paypal will consider it invalid due to "Duplicate txn_id", where the only previous records of the same transaction ID are the failed postbacks. I think this is a bug with duplicate_txn_id not filtering out flagged IPNs (they all have the same status, so it thinks they're duplicates), but maybe this is desired behaviour and I'm missing something.

The fact that PayPal sometimes repeats the failed IPNs and sometimes doesn't suggests that sometimes PayPal thinks it's confirmed the postback when giving the error, but I have no idea what could cause that.

Any suggestions? We've been banging our heads against the wall here.

The text was updated successfully, but these errors were encountered:

spookylukey · 2015-11-08T20:10:29Z

I haven't seen this, but I don't generally deal with a large volume of IPNs.

It would help if you gave more info about where this error message is appearing - there are multiple parts of the process where an error can occur. I would also create a fork and add logging calls until you can give additional info.

spookylukey · 2015-11-08T20:11:10Z

It does sound like a bug regarding duplicate transactions IDs, please file a separate issue/PR about that, thanks.

rakiru · 2015-11-08T22:32:48Z

From what I can gather, the IPN postback is returning that HTML page rather than "VERIFIED" or "INVALID". I don't have any other information since we can't reliably reproduce it, and i had to focus on the deadline. I'll try to get a minimal example set up with it happening and add heavy logging.

I mostly made this ticket to see if it was a known issue, since the (relatively few) other mentions of this on StackOverflow and such have no additional information. It looks like a generic error page (I've had the exact same response in my browser while trying to pay for something before), so it seems to be a bug on PayPal's end, but with so few Google results, it seems like something must trigger it. I understand if you want to close this issue since it's potentially not a django-paypal problem at all, but if I find out anything more, I'll post it here.

spookylukey · 2016-02-12T15:08:02Z

I'm going to close this ticket, but please do post more info and re-open if you can provide something that will enable django-paypal to behave better in this situation.

If we are hitting an error condition on PayPal, I think the best thing to do is leave the behaviour as it is - there isn't really anything we can do, and users of django-paypal need to be alerted to the problem.

Thanks!

Cojomax99 · 2016-02-12T15:12:12Z

This issue apparently is a known PayPal issue. See: http://stackoverflow.com/questions/33556807/django-paypal-ipn-requests-with-invalid-postback

Hopefully it will be fixed soon, but I am not holding out too much hope.

spookylukey · 2016-03-30T11:15:57Z

See 79b2227 which may help with part of this.

newearthmartin · 2019-07-03T20:08:16Z

Hi I'm experiencing the same. It is super minor, because Ive seen only 2 in 1500+ IPNs. But these two happened this week.
There is a new answer in the previously mentioned stackoverflow post:

You must reflect (postback) all parameters that Paypal sent you, untouched. Without messing up the encoding. Even if odd characters are occasionally in the data, like \ backslash. Accidentally corrupting the data will yield the "Invalid postback ... Fatal Failure" that you observed.

For detailed troubleshooting please see PP_MTS_Chad's excellent advice in notify_url never call when buyer paid for subscription

Maybe this happens in python 3 ? I changed only a few months ago. According to the above answer, it sounds like an encoding problem. Has it been solved in version 1.0.0 ? I'm using 0.5.0.

spookylukey · 2019-07-04T03:54:48Z

@newearthmartin it;s unlikely to be different in 1.0.0. If you could capture the exact, raw payload that causes this we might be able to debug it. Most likely it is some encoding issue and that makes it harder because getting the raw payload can be hard.

newearthmartin · 2019-07-11T09:32:14Z

Do you mean to capture the raw request headers and body received in the def ipn(request): view method?
I can do that and post it here once I get the error

newearthmartin · 2019-07-12T11:39:03Z

Hi @spookylukey !
I captured an IPN that causes this error
https://pastebin.com/raw/EWV5FDaZ

vmspike · 2019-07-14T16:50:16Z

@newearthmartin, I guess trailing s in request.body is a typo?
At least I cannot see any encoding (ascii vs windows-1251 vs unicode) issue for provided example, while windows-1251 usage for decoding/encoding looks like more correct choice than ascii because paypal use it.

newearthmartin · 2019-07-14T17:09:15Z

(Yes it was a typo, I updated the pastebin)
I'm taking a look at the code. This error is set in this method:

    def _verify_postback(self):
        if self.response != "VERIFIED":
            self.set_flag("Invalid postback. ({0})".format(self.response))

which is called from this method:

    def verify(self):
        """
        Verifies an IPN and a PDT.
        Checks for obvious signs of weirdness in the payment and flags appropriately.
        """
        self.response = self._postback().decode('ascii')
        self.clear_flag()
        self._verify_postback()
        ...

And I see that I this _postback() actually calls paypal

    def _postback(self):
        """Perform PayPal Postback validation."""
        return requests.post(self.get_endpoint(), data=b"cmd=_notify-validate&" + self.query.encode("ascii")).content

Previously I pasted the first request, I now understand that I need to capture the payload of this POST to paypal. I will add code to capture it, stay tuned!

newearthmartin · 2019-07-14T18:04:59Z

Ok, so I see that

# The following works if paypal sends an ASCII bytestring, which it does.
self.query = request.body.decode('ascii')

then we post it back

def _postback(self):
    """Perform PayPal Postback validation."""
    return requests.post(self.get_endpoint(), data=b"cmd=_notify-validate&" + self.query.encode("ascii")).content

So I think that the only thing that could go wrong is that something goes missing in the decode('ascii') encode('ascii') sequence. But I tried it with the actual request that gave an error and decode/encode produces the same original bytes. Other than that , it maybe is a bug in paypal? What do you guys think?

vmspike · 2019-07-15T08:47:50Z

Can be PayPal issue.
It's also better to encode/decode using windows-1252 instead of ascii (see charset=windows-1252 in request), but it's not the case for failed requests we're talking about.

newearthmartin · 2019-07-15T09:12:48Z

Would it be possible to store the request body plain and not decode/encode, and just attach it as is in the postback?

BTW, the encoding is a setting in PayPal interface https://pasteboard.co/Io4vJmW.png

spookylukey · 2019-07-15T19:34:28Z

The encode('ascii') is needed because we take the raw bytes from the HTTP request body (which are URL encoded, and therefore limited to ascii characters), and handle them internally as unicode characters, saving in the DB in a field that supports unicode etc.

We do this here:

https://github.com/spookylukey/django-paypal/blob/master/paypal/standard/models.py#L392

If there were non-ascii chars in the body, we'd be seeing exceptions here and we are not.

I'm not aware of an easy way to store raw bytes in a cross-DB safe way, that's why it's a unicode field.

We then need to convert this unicode object back to bytes to do a requests.post call.

So I don't think that changing the encode parameter will make a difference here. The windows-1252 charset is needed only when decoding the items from the body itself - for example if we had this in the body:

charset=windows-1252&foo=%E9

Then foo would have the value

urllib.parse.unquote_to_bytes('%E9').decode('windows-1252')

which is é. We are already doing this here - https://github.com/spookylukey/django-paypal/blob/master/paypal/standard/models.py#L247

That is a different level of encoding and decoding, which we don't need here.

My best guess at the moment is this is a spasmodic PayPal bug.

astrikov-d · 2019-07-18T01:30:52Z

We experiencing same issue in our website. This seems like a PayPal bug, since when we are resending IPN to our backend (emulating PayPal behaviour), it's being processed correctly and PayPal NVP api verifies this IPN as expected. We've started experiencing such errors from 11 July 2019 or so. I've already made a ticket in PayPal support but I don't think that they will do anything with it.
Currently the more or less simpliest solution is to remove or change transaction id of IPN that was not processed correctly and then resend IPN to PayPal listener in your django application.

spookylukey · 2019-07-18T08:25:16Z

Actually, there is a way to store binary data directly in Django, https://docs.djangoproject.com/en/2.2/_modules/django/db/models/fields/#BinaryField which has been around since 1.6 (after django-paypal was created). However, it comes with some disadvantages, being harder to read (as arbitrary binary data there is no obvious way to display it, despite the fact we know it can only be ascii chars), and we'd need migrations etc., and test them across multiple DBs. Plus I'm pretty sure this has nothing to do with the issue we are seeing, so there wouldn't seem to be any benefit in using it.

spookylukey · 2019-07-18T08:28:24Z

@astrikov-d Thanks for looking at this. I agree it is most likely a PayPal bug. We did think before about the possibility of automatically retrying if you get failure the first time, but I have no idea if that would help, and seems to introduce more uncertainty into our handling of it.

Would some manual but easy method to retry might be a benefit - for example, a "retry" button on the Django admin page for failed IPNs perhaps? I'm not sure that is possible, but could be helpful.

newearthmartin · 2019-07-18T08:31:27Z

I'm agreeing that it looks like a PayPal bug.

That said, storing in the db would not be necessary only a property in the ipn object that doesn't need to be persisted.

vorwrath · 2019-07-19T23:58:30Z

It seems like PayPal's response to the postback is supposed to have an HTTP status code of 200, and a body of "VERIFIED" or "INVALID". So what about immediately retrying if the status code of the response is not 200? Maybe make 3 attempts and then log the invalid postback the same as it does now if none are successful.

spookylukey · 2019-07-20T18:18:15Z

@vorwrath if these responses are returning non 200 responses that does sound like a reasonable solution, I'd accept a patch for that.

spookylukey · 2019-07-20T18:21:16Z

See also #216

ZackPlauche · 2021-07-17T10:37:32Z

I'm actually getting this error consistently. Mine doesn't say failure though, but I keep getting this one even though the transaction is completed. I haven't had 1 that wasn't flagged or Invalid yet. My invalid ipn signal is the only one that's been firing. I've tried this on Windows and Heroku.

EDIT: It randomly worked once, but after it stopped again.

Response:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>

Flag issue:

Invalid postback. (<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
)

Settings

PAYPAL_TEST = True

View

def paypal_form(request):
    host = request.get_host()
    payment_info = {
        'business': settings.PAYPAL_RECEIVER_EMAIL,
        'amount': '20',
        'currency_code': 'USD',
        'item_name': 'Test Item',
        'invoice': str(uuid.uuid4()),
        'notify_url': 'http://{}{}'.format(host,reverse('paypal-ipn')),
        'return_url': 'http://{}{}'.format(host,reverse('paypal-return')),
        'cancel_return': 'http://{}{}'.format(host,reverse('paypal-cancel')),
    }
    form = PayPalPaymentsForm(initial=payment_info)
    context = {
        'form': form,
    }
    return render(request, 'cax/paypal_form.html', context

Signals

@receiver(valid_ipn_received)
def payment_nofication_valid(sender, **kwargs):
    print('Valid ipn')

@receiver(invalid_ipn_received)
def payment_notification_invalid(sender, **kwargs):
    print ('invalid ipn')
    ipn = sender
    if ipn.payment_status == 'Completed':
        print('payment complete')

cankasap · 2021-08-01T14:55:38Z

Hello everyone,

@spookylukey - This topic is quite old but still open. I am getting %95 400 Bad and %5 200 OK from paypal.

I tried @vorwrath solution by adding a while loop in _postback function. 4 tries in total with 5 seconds interval.

My results:

Roughly;

% 55 of the postback returned OK in the second try.

% 25 of the postback returned OK in the third try.

The rest % 20 returned OK in the fourth or fifth, rarely returned 400 maybe %1-3.

Code is here:

def _postback(self):
    """Perform PayPal Postback validation."""
    x = requests.post(self.get_endpoint(), data=b"cmd=_notify-validate&" + self.query.encode(self.charset))
    print(x.encoding, x.reason, "context x")
    e = 0
    if x.reason == "Bad Request":
        while e < 4:
            time.sleep(5)
            x = requests.post(self.get_endpoint(), data=b"cmd=_notify-validate&" + self.query.encode(self.charset))
            print(x.encoding, x.reason, "context x")
            if x.reason == "OK":
                break
            print(e)
            e = e + 1
    return x.content

What are your toughts about this solution? Do you think it is reliable ? Why PayPal is doing this ?

spookylukey · 2021-08-01T19:12:47Z

If you could post a full, raw dump of the requests/responses involved, that might help debug this.

However, if you are getting success after repeating exactly the same request, that must be a PayPal bug AFAICS. So I think you should be reporting this to PayPal.

As for doing retries, I'm open to making this easier. In general, adding time.sleep feels like a bad solution - it will tie up your webserver process for all that time, which is asking for problems with availability. If we could find an async way to do this, it would be much better.

PSzczepanski1996 · 2021-08-03T12:31:25Z

I can confirm this bug. I get it inconsistently when using my own implementation using Paypal Smart buttons + paypal IPN setup at my own side, using the new API. I don't also get why the IPN examples are archived on PayPal github. The whole guide is an big mess, and I hate working with that. Even my-country (I'm from Poland) payment providers have SO BETTER integration. WHAT A BIG MESS.

I get paypal 400 error only when charset in request in IPN is set to iso-8859-1 despiteutf-8 is forced in IPN setting and it's just dice roll, sometimes it's utf-8, sometimes it's iso-8859-1.

I assume it can be because paypal uses load balancer in part of connection to servers, which connects sometimes to properly configured, and sometimes to invalid configured machine. I don't have other idea why it can be wrong.

cankasap · 2021-08-04T05:09:06Z

I am writing this to PayPal.

I do not know if below request response example can help but as @HoshiYamazaki said, if the response comes with iso-8859-1 encoding, it is a bad request. As you can see, 7th request came UTF-8 and went through no problem. Code is same as in previous post; a loop of 5 sec.

Django-Paypal is already taking care of this problem by using QueryDict so I dont know what is the solution.

try:
    data = QueryDict(request.body, encoding=encoding).copy()
except LookupError:
   ...
class QueryDict(MultiValueDict):
    ...
    if isinstance(query_string, bytes):
        # query_string normally contains URL-encoded data, a subset of ASCII.
        try:
            query_string = query_string.decode(self.encoding)
        except UnicodeDecodeError:
            # ... but some user agents are misbehaving :-(
            query_string = query_string.decode('iso-8859-1')

Maybe it is just Sandbox issue, may not happen in production. There is a post below if you have not seen it...

https://community.jboss.org/thread/38625

request: {'method': 'POST', 'url': 'https://ipnpb.sandbox.paypal.com/cgi-bin/webscr', 'headers': {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '/', 'Connection': 'keep-alive', 'Content-Length': '808'}, '_cookies': <RequestsCookieJar[]>, 'body': b'cmd=_notifyvalidate&mc_gross=282.70&invoice=---&protection_eligibility=Eligible&payer_id=---&payment_date=21%3A13%3A17+Aug+03%2C+2021+PDT&payment_status=Completed&charset=UTF8&first_name=John&mc_fee=8.50&notify_version=3.9&custom=&payer_status=verified&business=---&quantity=1&verify_sign=--.&payer_email=---&txn_id=---&payment_type=instant&last_name=Doe&receiver_email=---&payment_fee=&shipping_discount=0.00&rec
eiver_id=---&insurance_amount=0.00&txn_type=web_accept&item_name=---&discount=0.00&mc_currency=CAD&item_number=&residence_country=CA&test_ipn=1&shipping_method=Default&transaction_subject=&payment_gross=&ipn_track_id=---', 'hooks': {'response': []}, '_body_position': None}

response: encoding: iso-8859-1 reason: Bad Request 2 trying
response: encoding: iso-8859-1 reason: Bad Request 3 trying
response: encoding: iso-8859-1 reason: Bad Request 4 trying
response: encoding: iso-8859-1 reason: Bad Request 5 trying
response: encoding: iso-8859-1 reason: Bad Request 6 trying

response: encoding: UTF-8 reason: OK 7 trying

PSzczepanski1996 · 2021-08-04T05:38:56Z

@cankasap Everything is clear, the paypal has problem with IPN endpoint:

They provided me alternative address which should work, please try it.
I don't quite get one thing, why there are similar issues releated with IPN on examples repo, but maybe they are not the same problem, and I misunderstand them.

cankasap · 2021-08-04T06:53:05Z

@HoshiYamazaki I cant thank you enough, it works !!! I had almost given up... Got 25 200 OK out of 25... for the first time :)

rakiru mentioned this issue Nov 8, 2015

Duplicate txn_id check doesn't filter out flagged IPNs #119

Closed

spookylukey closed this as completed Feb 12, 2016

newearthmartin mentioned this issue Jul 14, 2019

Occasional invalid postbacks: 400 Bad Request #216

Closed

spookylukey reopened this Jul 20, 2019

spookylukey changed the title ~~Fatal Failure~~ Fatal Failure / 400 Bad Request Jul 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fatal Failure / 400 Bad Request #117

Fatal Failure / 400 Bad Request #117

rakiru commented Nov 5, 2015

spookylukey commented Nov 8, 2015

spookylukey commented Nov 8, 2015

rakiru commented Nov 8, 2015

spookylukey commented Feb 12, 2016

Cojomax99 commented Feb 12, 2016

spookylukey commented Mar 30, 2016

newearthmartin commented Jul 3, 2019 •

edited

Loading

spookylukey commented Jul 4, 2019

newearthmartin commented Jul 11, 2019

newearthmartin commented Jul 12, 2019 •

edited

Loading

vmspike commented Jul 14, 2019 •

edited

Loading

newearthmartin commented Jul 14, 2019 •

edited

Loading

newearthmartin commented Jul 14, 2019

vmspike commented Jul 15, 2019

newearthmartin commented Jul 15, 2019 •

edited

Loading

spookylukey commented Jul 15, 2019 •

edited

Loading

astrikov-d commented Jul 18, 2019

spookylukey commented Jul 18, 2019

spookylukey commented Jul 18, 2019

newearthmartin commented Jul 18, 2019

vorwrath commented Jul 19, 2019

spookylukey commented Jul 20, 2019

spookylukey commented Jul 20, 2019

ZackPlauche commented Jul 17, 2021 •

edited

Loading

cankasap commented Aug 1, 2021

spookylukey commented Aug 1, 2021

PSzczepanski1996 commented Aug 3, 2021 •

edited

Loading

cankasap commented Aug 4, 2021

PSzczepanski1996 commented Aug 4, 2021

cankasap commented Aug 4, 2021

Fatal Failure / 400 Bad Request #117

Fatal Failure / 400 Bad Request #117

Comments

rakiru commented Nov 5, 2015

spookylukey commented Nov 8, 2015

spookylukey commented Nov 8, 2015

rakiru commented Nov 8, 2015

spookylukey commented Feb 12, 2016

Cojomax99 commented Feb 12, 2016

spookylukey commented Mar 30, 2016

newearthmartin commented Jul 3, 2019 • edited Loading

spookylukey commented Jul 4, 2019

newearthmartin commented Jul 11, 2019

newearthmartin commented Jul 12, 2019 • edited Loading

vmspike commented Jul 14, 2019 • edited Loading

newearthmartin commented Jul 14, 2019 • edited Loading

newearthmartin commented Jul 14, 2019

vmspike commented Jul 15, 2019

newearthmartin commented Jul 15, 2019 • edited Loading

spookylukey commented Jul 15, 2019 • edited Loading

astrikov-d commented Jul 18, 2019

spookylukey commented Jul 18, 2019

spookylukey commented Jul 18, 2019

newearthmartin commented Jul 18, 2019

vorwrath commented Jul 19, 2019

spookylukey commented Jul 20, 2019

spookylukey commented Jul 20, 2019

ZackPlauche commented Jul 17, 2021 • edited Loading

cankasap commented Aug 1, 2021

spookylukey commented Aug 1, 2021

PSzczepanski1996 commented Aug 3, 2021 • edited Loading

cankasap commented Aug 4, 2021

PSzczepanski1996 commented Aug 4, 2021

cankasap commented Aug 4, 2021

newearthmartin commented Jul 3, 2019 •

edited

Loading

newearthmartin commented Jul 12, 2019 •

edited

Loading

vmspike commented Jul 14, 2019 •

edited

Loading

newearthmartin commented Jul 14, 2019 •

edited

Loading

newearthmartin commented Jul 15, 2019 •

edited

Loading

spookylukey commented Jul 15, 2019 •

edited

Loading

ZackPlauche commented Jul 17, 2021 •

edited

Loading

PSzczepanski1996 commented Aug 3, 2021 •

edited

Loading