EN-3146 Fix S3 upload retry

[stepan-mt]

EN-3146

Objective

Fix occasional tileset upload fails, namely RuntimeError: (400, b'') error response to HTTP PUT request when some part/chunk of the ingested file is uploaded to S3 storage.

Description

This error happened on servers belonging to Data team. They made a workaround based on containerization of maptiler-cloud-cli. Further investigation showed that invoking HTTP request to s3 storage (https://s3.gra.io.cloud.ovh.net) resolves to IPv6 address when maptiler-cloud-cli is invoked directly. On the other hand, it is resolved to IPv4 address when the tool is run from Docker. As a temporal workaround, Data team added entry in /etc/hosts file to always use IPv4.

However I was not able to reproduce the issue from my machine with IPv6 address, so I concentrated more on the code itself.

The former upload_to_s3 function has used optimized memoryview for upload parts/chunks. The advantage of this was that there is just one memory allocation for whole ingested file. However when there is urllib error that happens after request was sent (e.g. when reading the response from the socket), retry mechanism (urllib3.util.retry.Retry) re-uses the already exhausted iterator created by former read(lenght: int) function.

The simplest way how to reproduce the error is simply send empty body:

            response = http.request(
                method="PUT",
                url=part.url,
                headers={"Content-Length": str(length)},
                body=b"",
            )

In another words, for some reason servers of our Data team has sometime problem reading the response from OVH s3 server when they are connected via IPv6.

I'm aware that current code is slightly less optimized, because there is 8MB memory allocation for each part/chunk, however same approach is also used in upload_to_google_drive().

Acceptance

I monkey patched the HTTPConnection.getresponse function:

def create_flaky_getresponse(original_method, failure_rate=0.25):
    def wrapper(self, *args, **kwargs):
        if random.random() < failure_rate:
            print("    Request sent! Simulating read timeout...")
            raise socket.timeout("timed out")

        print("    Request sent! Response received...")
        return original_method(self, *args, **kwargs)

    return wrapper

original = HTTPConnection.getresponse
HTTPConnection.getresponse = create_flaky_getresponse(original)

Old version of the code:

$ poetry run maptiler-cloud --token XXX tiles ingest ~/Downloads/geo/grandcanyon-highres.png
Starting
Uploading
  part 1/19
    Request sent! Response received...
  part 2/19
    Request sent! Response received...
  part 3/19
    Request sent! Response received...
  part 4/19
    Request sent! Response received...
  part 5/19
    Request sent! Response received...
  part 6/19
    Request sent! Response received...
  part 7/19
    Request sent! Response received...
  part 8/19
    Request sent! Simulating read timeout...
    Request sent! Response received...
Traceback (most recent call last):
  File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/bin/maptiler-cloud", line 6, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
                                                                                               return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke                                                                                               return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                               File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/core.py", line 783, in invoke                                                                                                return __callback(*args, **kwargs)                                                                                                                                                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                           File "/home/stepo/.cache/pypoetry/virtualenvs/maptiler-cloud-cli-e1mT99qH-py3.12/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func                                                                                         return f(get_current_context(), *args, **kwargs)                                                                                                                                                                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                             File "/home/stepo/mt/maptiler-cloud-cli/src/maptiler/cloud_cli/base.py", line 227, in ingest_datasets                                                                                                                                            upload_result = upload_to_s3(container, ingest.upload)                                                                                                                                                                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                       File "/home/stepo/mt/maptiler-cloud-cli/src/maptiler/cloud_cli/base.py", line 321, in upload_to_s3                                                                                                                                               raise RuntimeError(response.status, response.read())                                                                                                                                                                                       RuntimeError: (400, b'')

New version of the code

$ poetry run maptiler-cloud --token XXX tiles ingest ~/Downloads/geo/grandcanyon-highres.png
Starting
Uploading
  part 1/19
    Request sent! Response received...
  part 2/19
    Request sent! Response received...
  part 3/19
    Request sent! Simulating read timeout...
    Request sent! Simulating read timeout...
    Request sent! Response received...
  part 4/19
    Request sent! Response received...
  part 5/19
    Request sent! Response received...
  part 6/19
    Request sent! Response received...
  part 7/19
    Request sent! Response received...
  part 8/19
    Request sent! Response received...
  part 9/19
    Request sent! Response received...
  part 10/19
    Request sent! Response received...
  part 11/19
    Request sent! Simulating read timeout...
    Request sent! Response received...
  part 12/19
    Request sent! Simulating read timeout...
    Request sent! Response received...
  part 13/19
    Request sent! Simulating read timeout...
    Request sent! Response received...
  part 14/19
    Request sent! Response received...
  part 15/19
    Request sent! Response received...
  part 16/19
    Request sent! Response received...
  part 17/19
    Request sent! Response received...
  part 18/19
    Request sent! Simulating read timeout...
    Request sent! Response received...
  part 19/19
    Request sent! Simulating read timeout...
    Request sent! Response received...
Processing
Finished
019ffb2b-28c4-7498-bfd7-d9d92c7328c1