Failed paid enrollments would never succeed

[arslanashraf7]

Steps to Reproduce

• Setup MITxOnline/edX
• Create some courses, enroll your user in a course, Audit Mode
• Don't create a paid mode for this course in edX yet
• Now in MITxOnline, Create a product for the course and try to enroll in Certificate/Paid track
• It would not succeed because there is no paid mode in edX
• Now create a paid mode Verified in edX Studio and try to run

from openedx.tasks import retry_failed_edx_enrollments
retry_failed_edx_enrollments()

• This would never succeed, Notice the stack trace. It complains about unauthorized to create enrollment in Verified mode

Problematic code
https://github.com/mitodl/mitxonline/blob/main/openedx/api.py#L683-L690
edx_client in this case would always be user-based instead of API KEY based. That's why it always fails.

Expected Behavior

• Retry task should fix the failed paid enrollments as well.

Actual Behavior

• Retry task never succeeds

Stacktrace

web_1       | [2023-08-02 07:29:50] ERROR 267 [courses.api] api.py:278 - [deec5b41c778] - edX enrollment failure for user: User username=arslan_02082023 [email protected], runs: ['course-v1:ARB+Chemistry_01+1']
web_1       | Traceback (most recent call last):
web_1       |   File "/app/./openedx/api.py", line 684, in enroll_in_edx_course_runs
web_1       |     enrollment = edx_client.enrollments.create_student_enrollment(
web_1       |   File "/usr/local/lib/python3.9/site-packages/edx_api/enrollments/__init__.py", line 165, in create_student_enrollment
web_1       |     resp.raise_for_status()
web_1       |   File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
web_1       |     raise HTTPError(http_error_msg, response=self)
web_1       | requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://host.docker.internal:18000/api/enrollment/v1/enrollment
web_1       | 
web_1       | The above exception was the direct cause of the following exception:
web_1       | 
web_1       | Traceback (most recent call last):
web_1       |   File "/app/./courses/api.py", line 264, in create_run_enrollments
web_1       |     enroll_in_edx_course_runs(
web_1       |   File "/app/./openedx/api.py", line 692, in enroll_in_edx_course_runs
web_1       |     raise EdxApiEnrollErrorException(user, course_run, exc) from exc
web_1       | openedx.exceptions.EdxApiEnrollErrorException: EdX API error enrolling user arslan_02082023 ([email protected]) in course run 'course-v1:ARB+Chemistry_01+1'.
web_1       | Response - code: 403, content: {"message":"User does not have permission to create enrollment with mode [verified]."}
celery_1    | [2023-08-02 07:29:50,464: INFO/MainProcess] Task ecommerce.tasks.send_ecommerce_order_receipt[c784dde8-2fd2-499d-9fea-1ae8f9b9d918] received
nginx_1     | "02/Aug/2023:07:29:50 +0000" client=172.18.0.9 method=POST request="POST /checkout/result/ HTTP/1.1" request_length=3452 status=302 bytes_sent=541 body_bytes_sent=0 referer=- user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/116.0" upstream_addr=172.18.0.7:8011 upstream_status=302 request_time=0.655 upstream_response_time=0.655 upstream_connect_time=0.000 upstream_header_time=0.655
web_1       | [pid: 267|app: 0|req: 16/21] 172.18.0.9 () {52 vars in 988 bytes} [Wed Aug  2 07:29:49 2023] POST /checkout/result/ => generated 0 bytes in 654 msecs (HTTP/1.1 302) 10 headers in 466 bytes (1 switches on core 16)