TLS/SSL asyncio leaks memory

[rojamit]

Bug report

Bug description:

python3.9 without uvloop doesn't leaks memory (or noticeably smaller).
python3.11+ (and others?) leaks memory A LOT under load (with or without uvloop) - up to +2gb per every test!

test commands:
ab -n50000 -c15000 -r https://127.0.0.1/
(apt install apache2-utils)

import asyncio, ssl, uvloop

class HTTP(asyncio.Protocol):
    
    def __init__(self):
        self.transport = None
        
    def connection_made(self, transport):
        self.transport = transport

    def data_received(self, data):
        self.transport.write(
            b'HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: keep-alive\r\n\r\n'
        )
        self.transport.close()
        
def make_tls_context():
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.load_cert_chain('cert.crt', 'cert.key')
    return ctx

tls_context = make_tls_context()
loop = uvloop.new_event_loop()

async def start_server(loop):
    return await loop.create_server(
        HTTP, '127.0.0.1', 443, backlog=65535,
        ssl=tls_context)

loop.run_until_complete(start_server(loop))
loop.run_forever()

CPython versions tested on:

3.9, 3.11, 3.12

Operating systems tested on:

Debian Linux

Linked PRs

• gh-109534: Asyncio sslproto memleak #113338
• gh-109534: fix reference leak when SSL handshake fails #114074
• gh-109534: switch from sock_call to sock_call_ex in sock_send #114311
• pythongh-109534: close transport if waiter is done #114367
• [3.12] gh-109534: fix reference leak when SSL handshake fails (GH-114074) #114829
• [3.11] gh-109534: fix reference leak when SSL handshake fails (GH-114074) #114830
• pythongh-109534: get rid of bytearray as class member #115643

[selevit]

Can confirm that the issue exists - Python 3.11, and uvloop 0.17.0, when trying to set up many client SSL connections that sometimes reconnects - even directly with asyncio.create_connection, and it's cleaned up not properly in some place.

Sometimes it leaked several megabytes per second for me.

I was able to track this by attaching to a running python process via gdb, and when it leaked, I put a catchpoint for brk syscalls, and examined 40 sequential breakpoint events, when all were related to /usr/lib/python3.10/ssl.py.

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.10/ssl.py", line 917, in read
    v = self._sslobj.read(len)
  <built-in method buffer_updated of uvloop.loop.SSLProtocol object at remote 0xffff758f90c0>
  <built-in method run of _contextvars.Context object at remote 0xffff758c7340>
  File "/home/ubuntu/py-core/env/lib/python3.10/site-packages/aiohttp/web.py", line 544, in run_app
    loop.run_until_complete(main_task)
  File "/home/ubuntu/py-core/server_app.py", line 39, in <module>
    web.run_app(app_factory(), loop=LOOP, host=args.host, port=args.port)
(gdb)

Also, a gc.get_objects() count was not actually growing, that's why I suppose that this is related a lower-level native implementation.

[arhadthedev]

@1st1, @asvetlov, @graingert, @gvanrossum, @kumaraditya303 (as asyncio module experts)

[gvanrossum]

Unfortunately I am really bad at this type of low level stuff. :-( If you suspect a leak in the C accelerator, you can disable it and see if the leak goes away. (Or at least becomes less severe.)

[gvanrossum]

OTOH if the problem is in the SSL code, we need a different kind of expert.

[gvanrossum]

Finally please figure out if this is in uvloop or not. If it’s tied to uvloop this is the wrong tracker.

[selevit]

Unfortunately, this bug was reproduced only in our production, and we weren't able to cause it in other environments.

It's already mitigated, as the root cause was in poor stability of connections.

I don't think I would be able to reproduce it again in production without uvloop, because it caused a lot of harm for our system.

But ready to assist and provide more details, if anyone wants to ask me some clarifying questions about it.

Also, there are several related tickets, and I'm not sure if they are all about uvloop:

1. asyncio: potential leak of TLS connections #106684 - very similar cpython issue, already closed though.
2. Memory leak with Python 3.11.2 aio-libs/aiohttp#7252 - also, an already closed, but recent aiohttp issue, they were able to mitigate it somehow, but not sure if it was the same bug.

[selevit]

I actually was able to reproduce this leak isolated several days ago, but it leaked for only ~5 megabytes in several hours. I adopted a repro script from this comment to the aiohttp bug mentioned above to be ready for run, and ran it for about 4 hours.

The initial memory consumption (RSS) had initially stabilised on 258120 bytes, but after several hours increased to ~264000. Hope it's the same bug.

#!/usr/bin/env python3
import aiohttp
import tracemalloc
import ssl
import asyncio

def init():
    timeout = aiohttp.ClientTimeout(connect=5, sock_read=5)
    ssl_ctx = ssl._create_unverified_context()
    conn = aiohttp.TCPConnector(ssl=ssl_ctx, enable_cleanup_closed=True)
    session = aiohttp.ClientSession(connector=conn, timeout=timeout, cookie_jar=aiohttp.CookieJar(unsafe=True))
    return session

async def fetch(client):
    try:
        async with client.request('GET', url=f'https://api.pro.coinbase.com/products/BTC-USD/ticker') as r:
            msg = await r.text()
            print(msg)
    except asyncio.CancelledError:
        raise
    except Exception as err:
        print("error", err)
        pass

async def main(): 
    requests = 600
    clients = [init() for _ in range(requests)]
    tracemalloc.start()
    try:
        while True:
            await asyncio.gather(*[fetch(client) for client in clients])
            await asyncio.sleep(5)
    except asyncio.CancelledError:
        pass  # end and clean things up
    finally:
        memory_used = tracemalloc.get_traced_memory()
        snapshot = tracemalloc.take_snapshot()
        stats = snapshot.statistics('lineno')
        for stat in stats[:10]:
            print(stat)
        try:
            await asyncio.gather(*[client.close() for client in clients.values()])
        except:
            pass

asyncio.run(main())

[gvanrossum]

Could you find a reproducer without aiohttp?

[Faolain]

Unsure if this is helpful at all or related but I came across this issue Kludex/uvicorn#2078 in which in the thread it is discussed/concluded that the issue of memory not being released is not isolated to uvicorn but seen in granian/gunicorn/hypercorn and as a result could be interpreter level (apologies for butchering the summary). Thread has some great charts/analysis though it is on the server level, however the different implementations w.r.t granian and uvicorn can help approximate where the issue might be surfacing if it's related. Example repo: https://github.com/Besedo/memory_issue/tree/main by @EBazarov Apologies in advance if it is unrelated and will remove the comment/create one in the right place.

gi0baro:

Given everything we stated in the upper replies, I suspect there's no much to do about this. Given the number of frameworks and servers tested, it just seems the Python interpreter is the culprit here, as even if the heap memory gets released, the RSS remains high.