uasyncio StreamWriter needs asyncio.sleep before running close() otherwise nothing is written to stream #11901

misaalanshori · 2023-06-29T13:30:03Z

misaalanshori
Jun 29, 2023

So I want to make a really simple and small HTTP server for the esp8266 using the uasyncio TCP server. But now I'm running into a problem where the server replies with nothing every time I make a request, and the solution that I found was to add an await asyncio.sleep(0.xx) before closing so it actually replies with the written data. here is the code that does the writing:

async def __handle_request(self, reader, writer):
        gc.collect()
        result = await self.__handle_route(await self.__process_header(reader))
        writer.write(f"HTTP/1.0 {result[0]} Works\r\n")
        writer.write(f"Content-Type: {result[1]}\r\n")
        writer.write(f"Content-Length: {len(result[2])}\r\n")
        writer.write(f"\r\n")
        writer.write(result[2])
        print("Sending: ", result[2])
        await writer.drain()
        await asyncio.sleep(0.5)
        writer.close()
        await writer.wait_closed()

Here, the longer it sleeps the result becomes more consistent, with smaller values sometimes it would give an empty reply and sometimes it would reply properly. But I feel like this isn't a good solution. I thought the await self.writer.drain() line would wait for all the written data to get written to the stream, but apparently not?

Are there any better solution for this?

Here is my micropython version:
MicroPython v1.20.0 on 2023-04-26; ESP module with ESP8266

Here is the full code:

import uasyncio as asyncio
import gc

loop = asyncio.get_event_loop()

def simpleHTML(text):
    return f'''<html><head><title>{text}</title></head><body><h1>{text}</h1></body></html>'''

class HttpServer:
    def __init__(self, host="0.0.0.0", port=80):
        self.host = host
        self.port = port
        self.routes = {}

    def start(self):
        loop.create_task(asyncio.start_server(self.__handle_request, self.host, self.port, 4096))
        loop.run_forever()

    def add_route(self, method, path):
        def decorator(callback):
            self.routes[(method, path)] = callback
            return callback
        return decorator

    async def __handle_route(self, request):
        print("Handling: ", request)
        if (request["method"], request["path"]) in self.routes:
            return await self.routes[(request["method"], request["path"])](request)
        else:
            return ["404", "text/html", simpleHTML("Not Found")]
    
    async def __process_header(self, reader):
        header = (await reader.readline()).decode().split(" ")
        path = header[1].split("?")
        query = {}
        if len(path) > 1:
            query = dict(q.split("=") for q in path[1].split("&"))
        return {"method": header[0], "path": path[0], "query": query}
    
    async def __handle_request(self, reader, writer):
        gc.collect()
        result = await self.__handle_route(await self.__process_header(reader))
        writer.write(f"HTTP/1.1 {result[0]} Works\r\n")
        writer.write(f"Content-Type: {result[1]}\r\n")
        writer.write(f"Content-Length: {len(result[2])}\r\n")
        writer.write(f"\r\n")
        writer.write(result[2])
        print("Sending: ", result[2])
        await writer.drain()
        await asyncio.sleep(0.5)
        writer.close()
        await writer.wait_closed()

Slightly unrelated note: I have tried other HTTP servers for micropython but they keep running out of memory, so I want to try to make my own. Though this one is still not small enough, my non-asynchronous version leaves me with 23kb of memory, and this one leaves me with 8kb of memory. Maybe uasyncio just needs a lot of memory?

stupid-stove · 2025-03-26T13:37:35Z

stupid-stove
Mar 26, 2025

I've just had the same issue with the sleep. It's just a wild guess, but maybe drain only flushes the data into the networking device. I.e. it doesn't actually send a TCP package for every drain (which would make sense). Then wait_closed closes the TCP connection before all data has been send to the peer. Waiting a bit would make the networking device send the actual packages???
Although this theory does not help finding a solution to the issue.

Did you find anything out about it?

0 replies

DavesCodeMusings · 2025-03-27T03:46:45Z

DavesCodeMusings
Mar 27, 2025

I have an HTTP web server / web framework I've developed. If you want to look it over for inspiration, it's here: https://github.com/DavesCodeMusings/thimble

It's been a while since I've run it on an 8266, but it did work on a NodeMCU 12E last I checked.

I haven't found the need to use sleep functions.

The connection handler looks like this:

async def on_connect(self, reader, writer):
"""
Connection handler for new client requests.

Args:
    reader (stream): input received from the client
    writer (stream): output to be sent to the client

Returns:
    nothing
"""
remote_addr = writer.get_extra_info('peername')[0]
if self.debug:
    print(f'Connection from client: {remote_addr}')

try:
    req_buffer = await reader.read(self.req_buffer_size)
    req = await Thimble.parse_http_request(req_buffer)
except Exception as ex:
    await self.send_error(400, writer)
    print(f'Unable to parse request: {ex}')
else:
    req['remote_addr'] = remote_addr
    if self.debug:
        print(f'Request: {req}')
    route_value = self.resolve_route(req['method'] + req['path'])
    if isinstance(route_value, tuple):  # a function and URL wildcard value were returned
        await self.send_function_results(route_value[0], req, route_value[1], writer)
    elif route_value is not None:  # just a function was returned
        await self.send_function_results(route_value, req, None, writer)
    else:  # nothing returned, try delivering static content instead
        file_path = self.static_folder + req['path']
        if file_path.endswith('/'):  # '/path/to/' becomes '/path/to/index.html'
            file_path = file_path + self.directory_index
        await self.send_file_contents(file_path, req, writer)

await writer.drain()
writer.close()
await writer.wait_closed()
reader.close()
await reader.wait_closed()
if self.debug:
    print(f'Connection closed for {remote_addr}')

The one thing I did find that really ate up memory was serving files from the flash file system. I had to create a function to serve them in 512 byte chunks, otherwise the device would run out of RAM because of buffering.

I'm also using pre-compiled bytecode with the thimble.mpy file. That may help ease memory usage as well.

Give my module a try if you want. Or keep hacking at your own, it can be a very rewarding experience. Just give me a shout if you have any questions.

2 replies

stupid-stove Mar 28, 2025

I've just tried your WS. It error's out, when accessing non routes (I assume this is not intended, so I've created an issue DavesCodeMusings/thimble#17).

Except that though, it does not reset the connection, so I will look into that later.

DavesCodeMusings Mar 28, 2025

I've just tried your WS. It error's out, when accessing non routes (I assume this is not intended, so I've created an issue DavesCodeMusings/thimble#17).

Fixed in commit DavesCodeMusings/thimble@4b6b122.

Thanks for the bug report.

DavesCodeMusings · 2025-03-27T03:48:53Z

DavesCodeMusings
Mar 27, 2025

Whoa. Just realized this thread is pretty old. Oh well, if it helps someone...

0 replies

peterhinch · 2025-03-27T12:32:37Z

peterhinch
Mar 27, 2025
Collaborator

The source indicates that drain does wait for the socket (s) to send the data - because it uses the return value of s.write(). There may be latency elsewhere but when drain returns the socket "thinks" it's sent all the data.

0 replies

stupid-stove · 2025-03-28T16:23:18Z

stupid-stove
Mar 28, 2025

Ok, I've read somewhere, that you should call drain, after every write. And I've read, that calling write multiple times before calling drain makes new allocations vs calling it 1-1. So I have a lot of small writes in my code. It does not like this. When writing the entire http message with one write, it works just fine, but when writing it byte by byte, we get a connection reset.

# ok
    async def __on_connect_test(self, r, w): 
        await r.readline()
        
        w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi')
        await w.drain()
        
        r.close()
        w.close()
        await r.wait_closed()
        await w.wait_closed()
    
   # connection reset
    async def __on_connect_test_short(self, r, w):
        await r.readline()
        m = b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi'
        
        for i in range(len(m)):
            w.write(m[i:i+1])
            await w.drain()
        
        r.close()
        w.close()
        await r.wait_closed()
        await w.wait_closed()

# ok
    async def __on_connect_test_short_sleep(self, r, w):
        await r.readline()
        m = b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi'
        
        for i in range(len(m)):
            w.write(m[i:i+1])
            await w.drain()
        
        r.close()
        w.close()
        await asyncio.sleep(2)
        await r.wait_closed()
        await w.wait_closed()

But why?

6 replies

stupid-stove Mar 29, 2025

So, ... the server actually never responded correctly. Looking at the network traffic we see that it always just sends a RESET instead of a proper TCP connection teardown, when it thinks its finished (i.e. calling wait_closed). Calling sleep allows the peer to receive the packages before the server just RESET's the connection. I don't think anymore that this had anything todo with the buffer length, as handle_client_one_looooong_write fails, but handle_client_one_write goes through.

I assume the first TCP packet will always go through. Which is the entire message in the case of handle_client_one_write (-> success), but not for handle_client_one_looooong_write (-> did not receive rest of message before RESET came in).

This seems to not be intended. I should probably open an issue.

I've only tested this on my W5500-EVB-Pico so far. It might (likely) be an issue specific to this device/driver. I'd like to try this on another device first to make sure.

I'll add a few TCP dumps showing this behaviour tcp dumps.zip.

Here's my full script for testing:

import network
from machine import SPI, Pin
import time
import asyncio

def w5x00_init():
    # https://github.com/Wiznet/RP2040-HAT-MicroPython/blob/main/Ethernet%20Example%20Getting%20Started%20%5BMicropython%5D.md
    spi=SPI(0,2_000_000, mosi=Pin(19),miso=Pin(16),sck=Pin(18))
    nic = network.WIZNET5K(spi,Pin(17),Pin(20))
    nic.active(True)
    nic.ifconfig('dhcp')
    while not nic.isconnected():
        time.sleep(1)
        print(nic.regs())
    print(nic.ifconfig())
    

# ok
async def handle_client_one_write(r, w):
    await r.readline()
    
    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

# connection reset
async def handle_client_one_byte_at_a_time(r, w):
    await r.readline()
    m = b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi'
    
    for i in range(len(m)):
        w.write(m[i:i+1])
        await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()
    
    
# ok
async def handle_client_one_byte_at_a_time_sleep_afterwards(r, w):
    await r.readline()
    m = b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi'
    
    for i in range(len(m)):
        w.write(m[i:i+1])
        await w.drain()
    
    r.close()
    w.close()
    await asyncio.sleep(2)
    await r.wait_closed()
    await w.wait_closed()


# receives parts of the message, then connection reset
'''
    Cmd (Windows): curl 192.168.2.127
    Output: hihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihihcurl: (56) Recv failure: Connection was reset
    
    Note: The output is not consisent, I've seen 1895-1989 chars being printed (limited testing)
'''
async def handle_client_one_byte_at_a_time_long_message(r, w):
    await r.readline()
    m = b'HTTP/1.0 200 OK\r\nContent-Length: 2000\r\nContent-Type: text/plain\r\n\r\n' + b'hi'*1000
    
    for i in range(len(m)):
        w.write(m[i:i+1])
        await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()


# connection reset
async def handle_client_one_byte_at_the_end(r, w):
    await r.readline()
    
    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nh')
    await w.drain()
    w.write(b'i')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

# partial send, then connection reset
async def handle_client_single_bytes_at_the_start(r, w):
    await r.readline()
    
    m = b'HTTP/1.0 200 OK\r\nContent-Length: 2000\r\nContent-Type: text/plain\r\n\r\n' + b'hi'*500
    for i in range(len(m)):
        w.write(m[i:i+1])
        await w.drain()
        
    w.write(b'hi'*500)
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

# connection reset
async def handle_client_single_byte_at_start_then_rest(r, w):
    await r.readline()
    w.write(b'H')
    await w.drain()
    w.write(b'TTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

# partial write, then connection reset.
async def handle_client_single_byte_at_start_then_rest_long(r, w):
    await r.readline()
    w.write(b'H')
    await w.drain()
    w.write(b'TTP/1.0 200 OK\r\nContent-Length: 2000\r\nContent-Type: text/plain\r\n\r\n'+1000*b'hi')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()


# connection reset
async def handle_client_three_packets(r, w):
    await r.readline()
    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 100')
    await w.drain()
    w.write(b'\r\nContent-Type: text/plain\r\n\r\n'+20*b'hi')
    await w.drain()
    w.write(30*b'hi')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

# partial write, reset
async def handle_client_three_packets_big(r, w):
    await r.readline()
    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 440\r\nContent-Type: text/plain\r\n\r\n'+40*b'hi')
    await w.drain()
    w.write(90*b'hi')
    await w.drain()
    w.write(90*b'hi')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()
    
# partial write, connection reset
async def handle_client_one_looooong_write(r, w):
    await r.readline()
    
    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 1000\r\nContent-Type: text/plain\r\n\r\n'+500*b'hi')
    await w.drain()
    
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

async def server(handle_client):
    w5x00_init()
    
    await asyncio.start_server(handle_client, '0.0.0.0', 80)
    
    while True:
        await asyncio.sleep(1)
    

asyncio.run(server(handle_client_one_looooong_write))

EDIT:
I've just tried the thimble webserver and it does a connection tairdown! But only one sided: it ACK's the FIN from the client but sends no own FIN.

Which would be a different result, but still not the expected result. dump: thimble.zip. --> There must still be some kind of software difference we are missing.

EDIT 2:
Just not closing the connection seems to work just fine. I.e. it does the one sided tairdown without sending RESET.

async def handle_client_one_write_no_close(r, w):
    await r.readline()
    
    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi')
    await w.drain()

stupid-stove Mar 31, 2025

I think I've found the source of when it sends a RESET vs. doing a partial connection tairdown.
When not reading everything from the reader it always does a RESET. In my implementation I often do an early return, since all I care about is the first line of the request. But in the Thimble webserver always reads self.req_buffer_size=1024 bytes, i.e. the entire message.

@DavesCodeMusings If I send send a message longer then the buffer you read e.g. GET /hello?foo=baaaaaaa... your webserver also does not do a proper connection tairdown, but sends a RESET. (Although in your case I'm not sure this is considered an issue, as you just don't handle requests longer then the buffer either way.)

But the really strange thing is, that when doing

async def handle_client_read_1000(r, w):
    await r.read(1000)

    w.write(b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi')
    r.close()
    w.close()
    await r.wait_closed()
    await w.wait_closed()

with a short request (curl 192.168.2.127) it does a proper connection teardown (i.e. 2-way)!!

I can't imagine why this would be intentional.

The last question is, wherever this is a problem with the W5500 or a generic micropython thing.
Sadly I've not been able to get my Pico W to work, so I cannot test this right now.

stupid-stove Mar 31, 2025

I've been able to get my Pico W running: On fist glances it's the same behaviour.

DavesCodeMusings Mar 31, 2025

@DavesCodeMusings If I send send a message longer then the buffer you read e.g. GET /hello?foo=baaaaaaa... your webserver also does not do a proper connection tairdown, but sends a RESET. (Although in your case I'm not sure this is considered an issue, as you just don't handle requests longer then the buffer either way.)

I don't think it's me sending the RESET. Maybe a low-level socket response to overflowing the buffer?

Below is the code in my connection handler. It should always reach the drain, close, wait_closed no matter what.

try:
    req_buffer = await reader.read(self.req_buffer_size)
    req = await Thimble.parse_http_request(req_buffer)
except Exception as ex:
    await self.send_error(400, writer)
    print(f'Unable to parse request: {ex}')
else:
    req['remote_addr'] = remote_addr
    if self.debug:
        print(f'Request: {req}')
    route_value = self.resolve_route(req['method'] + req['path'])
    if isinstance(route_value, tuple):  # a function and URL wildcard value were returned
        await self.send_function_results(route_value[0], req, route_value[1], writer)
    elif route_value is not None:  # just a function was returned
        await self.send_function_results(route_value, req, None, writer)
    else:  # nothing returned, try delivering static content instead
        file_path = self.static_folder + req['path']
        if file_path.endswith('/'):  # '/path/to/' becomes '/path/to/index.html'
            file_path = file_path + self.directory_index
        await self.send_file_contents(file_path, req, writer)

await writer.drain()
writer.close()
await writer.wait_closed()
reader.close()
await reader.wait_closed()

stupid-stove Apr 1, 2025

Reaching the drain/close/wait is not the issue.
If I run

curl 192.168.2.127/hiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

on your server, I get

curl: (56) Recv failure: Connection was reset

(Sometimes we also get a 400 Bad Request. I think this just depends, on which packet the client receives first. No. 8 is the Bad Request response)

As you can see here

your webserver sends the RESET, after it sends the Bad Request response. It does not do a TCP connection teardown.

"Maybe a low-level socket response to overflowing the buffer?" - I can't say, WHY this happens, but by now I'm relatively confident I know, WHEN it happens, i.e. not reading the entire buffer. But I don't think there can by any overflowing or any thing like that, because, we transmit the same amount of data every time. The entire message has to be in the buffer either way, as we are reading it in one go.

Maybe @peterhinch can find the part in the repo where we read from the TCP buffer (as I am sadly not familiar with this repo) and we can go from there.

Note: just a sanity check, but I've validated, that CPython does a proper connection teardown in this scenario.

Here the full webserver for reference

import network
from machine import SPI, Pin
import time

from asyncio import get_event_loop, start_server
from os import stat
from re import search

class Thimble:
    """
    A tiny web framework in the spirit of Flask, scaled down to run on microcontrollers
    """
    def __init__(self, default_content_type='application/octet-stream', req_buffer_size=1024):
        self.routes = {}  # Dictionary to map method and URL path combinations to functions
        self.default_content_type = default_content_type
        self.req_buffer_size = req_buffer_size
        self.static_folder = '/static'
        self.directory_index = 'index.html'
        self.error_text = {
            400: "400: Bad Request",
            404: "404: Not Found",
            500: "500: Internal Server Error"
        }
        self.media_types = {
            'css': 'text/css',
            'html': 'text/html',
            'ico': 'image/vnd.microsoft.icon',
            'jpg': 'image/jpeg',
            'js': 'text/javascript',
            'json': 'application/json',
            'otf': 'font/otf',
            'png': 'image/png',
            'svg': 'image/svg+xml',
            'ttf': 'font/ttf',
            'txt': 'text/plain',
            'woff': 'font/woff',
            'woff2': 'font/woff2'
        }

    server_name = 'Thimble (MicroPython)'  # Used in 'Server' response header.

    @staticmethod
    def parse_query_string(query_string):
        """
        Split a URL's query string into individual key/value pairs
        (ex: 'pet=Panda&color=Red' becomes { "pet": "panda", "color": "red"}
        Args:
            query_string (string): the query string portion of a URL (without the leading ? delimiter)

        Returns:
            dictionary: key/value pairs
        """
        query = {}
        query_params = query_string.split('&')
        for param in query_params:
            if '=' not in param:  # A key with no value, like: 'red' instead of 'color=red'
                key = param
                query[key] = ''
            else:
                key, value = param.split('=')
                query[key] = value

        return query

    @staticmethod
    async def parse_http_request(req_buffer):
        """
        Given a raw HTTP request, return a dictionary with individual elements broken out

        Args:
            req_buffer (bytes): the unprocessed HTTP request sent from the client

        Raises:
            exception: when the request buffer is empty

        Returns:
            dictionary: key/value pairs including, but not limited to method, path, query, headers, body, etc.
                or None type if parsing fails
        """
        assert (req_buffer != b''), 'Empty request buffer.'

        req = {}
        req_buffer_string = req_buffer.decode('utf8')
        req_buffer_lines = req_buffer_string.split('\r\n')
        del req_buffer_string  # free up for garbage collection

        req['method'], target, req['http_version'] = req_buffer_lines[0].split(
            ' ', 2)  # Example: GET /route/path HTTP/1.1
        if '?' not in target:
            req['path'] = target
        else:  # target can have a query component, so /route/path could be something like /route/path?state=on&timeout=30
            req['path'], query_string = target.split('?', 1)
            req['query'] = Thimble.parse_query_string(query_string)

        req['headers'] = {}
        for i in range(1, len(req_buffer_lines) - 1):
            # Blank line signifies the end of headers.
            if req_buffer_lines[i] == '':
                break
            else:
                name, value = req_buffer_lines[i].split(':', 1)
                req['headers'][name.strip().lower()] = value.strip()

        # Last line is the body (or blank if no body.)
        req['body'] = req_buffer_lines[len(req_buffer_lines) - 1]

        return req

    @staticmethod
    async def http_status_line(status_code):
        """
        Given an HTTP status code (e.g. 200, 404, etc.), format the server response status line

        Args:
            status_code (int): the HTTP status code as defined by RFC 7231 Respone Status Codes (ex. 200)

        Returns:
            string: HTTP status line with protocol version, numeric status code, and corresponding status text
        """
        http_status_message = {
            200: "200 OK",
            302: "302 Found",
            400: "400 Bad Request",
            404: "404 Not Found",
            500: "500 Internal Server Error"
        }

        if status_code is None or status_code not in http_status_message:
            status_code = 500

        return f'HTTP/1.1 {http_status_message[status_code]}\r\n'

    @staticmethod
    async def http_headers(content_length=0, content_type='', content_encoding=''):
        """
        Generate appropriate HTTP response headers based on content properties

        Args:
            content_length (int): length of body, used for Content-Length header
            content_type (string): media-type of body, used for Content-Type header
            content_encoding(string): compression type, used for Content-Encoding header

        Returns:
            string: HTTP headers separated by \r\n
        """
        headers = 'Connection: close\r\n'
        if content_encoding != '':
            headers += f'Content-Encoding: {content_encoding}\r\n'
        headers += f'Content-Length: {content_length}\r\n'
        if content_type != '':
            headers += f'Content-Type: {content_type}\r\n'
        headers += f'Server: {Thimble.server_name}\r\n'
        headers += '\r\n'  # blank line signifies end of headers

        return headers

    # MicroPython does not have a built-in types module, so to avoid external dependencies this method lets
    # Thimble determine if route functions should be awaited or not by comparing their type to known async
    # and regular functions that are already part of the code.

    @staticmethod
    def is_async(func):
        """
        Determine if a function is async not by comparing its type to known async and regular functions.

        Args:
            func (object): a reference to the function being examined

        Returns:
            boolean: True if the function was defined as asynchronous, False if not, and None if unknown
        """
        if type(func) == type(Thimble.on_connect):  # noqa: E721
            return True  # It's an async function
        elif type(func) == type(Thimble.run):  # noqa: E721
            return False  # It's a regular function
        else:
            return None  # It's not a function

    async def send_error(self, error_number, writer):
        """
        Given a stream and an HTTP error number, send a friendly text error message.

        Args:
            error_number (integer): HTTP status code
            writer (object): the asyncio Stream object to which the file should be sent

        Returns:
            nothing
        """
        writer.write(await Thimble.http_status_line(error_number))
        error_text = f'{self.error_text[error_number]}\r\n'
        writer.write(await Thimble.http_headers(content_type='text/plain', content_length=len(error_text)))
        writer.write(error_text)
        await writer.drain()

    async def send_function_results(self, func, req, url_wildcard, writer):
        """
        Execute the given function with the HTTP reqest parameters as an argument and send the results as an HTTP reply

        Args:
            func (object): reference to the function to be executed or a tuple of function and URL wildcard
            req (dictionary): HTTP request parameters
            url_wildcard (various types): regex-matched portion of the url_path (or None for non-regex routes)
            writer (object): the asyncio Stream object to which the results should be sent

        Returns:
            nothing
        """
        try:
            if Thimble.is_async(func) is True:  # await the async function
                if url_wildcard is not None:
                    func_result = await func(req, url_wildcard)
                else:
                    func_result = await func(req)
            else:  # no awaiting required for non-async
                if url_wildcard is not None:
                    func_result = func(req, url_wildcard)
                else:
                    func_result = func(req)

        except Exception as ex:
            await self.send_error(500, writer)
            print(f'Function call failed: {ex}')
        else:
            if isinstance(func_result, tuple) and len(func_result) == 3:
                body, status_code, content_type = func_result
            elif isinstance(func_result, tuple) and len(func_result) == 2:
                body, status_code = func_result
                content_type = 'text/plain'
            else:
                body = func_result
                status_code = 200
                content_type = 'text/plain'

            if not isinstance(body, str):
                body = str(body)
            writer.write(await Thimble.http_status_line(status_code))
            writer.write(await Thimble.http_headers(content_length=len(body), content_type=content_type))
            await writer.drain()
            writer.write(body)
            await writer.drain()

    @staticmethod
    async def file_size(file_path):
        """
        Given a path to a file, return the file's size or None if there's an exception when checking.

        Args:
            file_path (string): a fully-qualified path to the location of the file

        Returns:
            file size in bytes or None if there was a problem obtaining the size (e.g. file does not exist)
        """

        try:
            size = stat(file_path)[6]
        except OSError:
            size = None

        return size

    async def file_type(self, file_path):
        """
        Return a standard media type / subtype based on file extension

        Args:
            file_path (string): file name or full path

        Returns:
            string: media type as registered with the Internet Assigned Numbers Authority (IANA)
        """
        file_ext = file_path.split('.')[-1]
        if file_ext not in self.media_types:
            return self.default_content_type
        else:
            return self.media_types[file_ext]

    @staticmethod
    def read_file_chunk(file):
        """
        Given a file handle, read the file in small chunks to avoid large buffer requirements.

        Args:
            file (object): the file handle returned by open()

        Returns:
            bytes: a chunk of the file until the file ends, then nothing
        """
        while True:
            chunk = file.read(512)  # small chunks to avoid out of memory errors
            if chunk:
                yield chunk
            else:  # empty chunk means end of the file
                return

    async def send_file_contents(self, file_path, req, writer):
        """
        Given a file path and an output stream, send HTTP status, headers, and file contents as body.
        If client accepts gzip encoding, and a file of the same name with a .gzip extension appended
        exists, the gzipped version will be sent. This is not so much for speeding up transfer as it
        for conserving limited flash filesystem space.

        Args:
            file_path (string): fully-qualified path to file
            writer (object): the asyncio Stream object to which the file should be sent

        Returns:
            nothing
        """
        # file_size is also used as an indicator of the file's existence
        file_gzip_size = await Thimble.file_size(file_path + '.gzip')
        file_size = await Thimble.file_size(file_path)
        file_type = await self.file_type(file_path)

        if file_gzip_size is not None and 'accept-encoding' in req['headers'] and 'gzip' in req['headers']['accept-encoding'].lower():
            writer.write(await Thimble.http_status_line(200))
            writer.write(await Thimble.http_headers(content_length=file_gzip_size, content_type=file_type, content_encoding='gzip'))
            with open(file_path + '.gzip', 'rb') as file:
                for chunk in Thimble.read_file_chunk(file):
                    writer.write(chunk)
                    await writer.drain()  # drain immediately after write to avoid memory allocation errors
        elif file_size is not None:  # a non-compressed file was found
            writer.write(await Thimble.http_status_line(200))
            writer.write(await Thimble.http_headers(content_length=file_size, content_type=file_type))
            with open(file_path, 'rb') as file:
                for chunk in Thimble.read_file_chunk(file):
                    writer.write(chunk)
                    await writer.drain()
        else:  # no file was found
            await self.send_error(404, writer)
            print(f'Error reading file: {file_path}')

    def route(self, url_path, methods=['GET']):
        """
        Given a URL path and list of zero or more HTTP methods, add the decorated function to the route table.

        Args:
            url_path (string): path portion of a URL (ex. '/path/to/thing') that will trigger a call to the function
            methods (list): a list of any HTTP methods (eg. ['GET', 'PUT']) that are used to trigger the call

        Returns:
            object: wrapper function
        """
        regex_macros = {
            '<digit>': '([0-9])',
            '<float>': '([-+]?[0-9]*\.?[0-9]+)',
            '<int>': '([0-9]+)',
            '<string>': '(.*)'
        }

        regex_match = search('(<.*>)', url_path)
        if regex_match:
            url_path = url_path.replace(regex_match.group(
                1), regex_macros[regex_match.group(1)])

        def add_route(func):
            for method in methods:
                # Methods are uppercase (see RFC 9110)
                self.routes[method.upper() + url_path] = func

        return add_route

    def resolve_route(self, route_candidate):
        """
        Given a route pattern (METHOD + url_path), look up the corresponding function.

        Args:
            route_candidate (string): An uppercase HTTP method concatenated with a URL path wich may contain regex (ex: GET/gpio/([0-9+])$)

        Returns:
            object: reference to function (for non-regex URLs) or tuple with function and regex capture (for regex URLs)
        """
        result = None
        if self.debug:
            print("Looking for a route matching:", route_candidate)
        if route_candidate in self.routes:  # pattern is a non-regex string, like: 'GET/gpio/2'
            if self.debug:
                print("Route table had an exact match. We're done!")
            result = self.routes[route_candidate]
        else:  # pattern may contain regex, like 'GET/gpio/([0-9]+)'
            if self.debug:
                print("No exact match in route table. Looking for regex matches...")
            for stored_route in self.routes.keys():
                if self.debug:
                    print("Examining this route table entry for potential match:", stored_route)
                regex_match = search("^" + stored_route + "$", route_candidate)
                if regex_match:
                    if self.debug:
                        print("Found a regex match with:", stored_route)
                    func = self.routes[stored_route]
                    wildcard_value = regex_match.group(1)
                    if self.debug:
                        print(f'Extracted wildcard value: {wildcard_value}')
                    result = func, wildcard_value
                else:
                    if self.debug:
                        print("No match.")

        return result

    async def on_connect(self, reader, writer):
        """
        Connection handler for new client requests.

        Args:
            reader (stream): input received from the client
            writer (stream): output to be sent to the client

        Returns:
            nothing
        """
        remote_addr = writer.get_extra_info('peername')[0]
        if self.debug:
            print(f'Connection from client: {remote_addr}')

        try:
            req_buffer = await reader.read(self.req_buffer_size)
            req = await Thimble.parse_http_request(req_buffer)
        except Exception as ex:
            await self.send_error(400, writer)
            print(f'Unable to parse request: {ex}')
        else:
            req['remote_addr'] = remote_addr
            if self.debug:
                print(f'Request: {req}')
            route_value = self.resolve_route(req['method'] + req['path'])
            if isinstance(route_value, tuple):  # a function and URL wildcard value were returned
                await self.send_function_results(route_value[0], req, route_value[1], writer)
            elif route_value is not None:  # just a function was returned
                await self.send_function_results(route_value, req, None, writer)
            else:  # nothing returned, try delivering static content instead
                file_path = self.static_folder + req['path']
                if file_path.endswith('/'):  # '/path/to/' becomes '/path/to/index.html'
                    file_path = file_path + self.directory_index
                await self.send_file_contents(file_path, req, writer)

        await writer.drain()
        writer.close()
        await writer.wait_closed()
        reader.close()
        await reader.wait_closed()
        if self.debug:
            print(f'Connection closed for {remote_addr}')

    def run(self, host='0.0.0.0', port=80, loop=None, debug=False):
        """
        Start an asynchronous listener for HTTP requests.

        Args:
            host (string): the IP address of the interface on which to listen
            port (int): the TCP port on which to listen
            loop (object): the asyncio loop that the server should insert itself into
            debug (boolean): a flag to indicate verbose logging is desired

        Returns:
            object: the same loop object given as a parameter or a new one if no existing loop was passed
        """
        self.debug = debug
        print(f'Listening on {host}:{port}')

        if loop is None:
            loop = get_event_loop()
            server = start_server(self.on_connect, host, port, 5)
            loop.create_task(server)
            loop.run_forever()
        else:
            server = start_server(self.on_connect, host, port, 5)
            loop.create_task(server)

        return loop
    

def w5x00_init():
    # https://github.com/Wiznet/RP2040-HAT-MicroPython/blob/main/Ethernet%20Example%20Getting%20Started%20%5BMicropython%5D.md
    spi=SPI(0,2_000_000, mosi=Pin(19),miso=Pin(16),sck=Pin(18))
    nic = network.WIZNET5K(spi,Pin(17),Pin(20))
    nic.active(True)
    nic.ifconfig('dhcp')
    while not nic.isconnected():
        time.sleep(1)
        print(nic.regs())
    print(nic.ifconfig())
    
w5x00_init()


from thimble import Thimble

app = Thimble() 

@app.route("/")
def hello(req):
    return "Hello World"

@app.route("/gpio/<digit>")
def gpio(req, num):
    return f"Hello GPIO {num}"

app.run(debug=True)

stupid-stove · 2025-04-02T18:00:26Z

stupid-stove
Apr 2, 2025

I've just checked, this is not an issue for just asyncio, but a general issue. When writing a synchronous server:

def server_sync():
    import socket
    listening_soc = socket.socket()
    listening_soc.bind(('0.0.0.0', 80))
    listening_soc.listen(1)
    while True:
        soc, _ = listening_soc.accept()
        
        r = soc.read(1)
        
        soc.write(b'HTTP/1.0 200 OK\r\nContent-Length: 2\r\nContent-Type: text/plain\r\n\r\nhi')
        soc.close()

It also RESET's the connection.

0 replies

uasyncio StreamWriter needs asyncio.sleep before running close() otherwise nothing is written to stream #11901

Uh oh!

Uh oh!

Replies: 6 comments · 8 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterhinch Mar 27, 2025 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 6 comments 8 replies

peterhinch
Mar 27, 2025
Collaborator