Memory is tight with v1.29.0 on ESP8266

[dave0003]

I upgraded one of my ESP8266's from v1.28.0 to v1.29.0, and stuff that worked before, won't work anymore. I get unable to allocate memory errors with v1.29.0 while v1.28.0 is still OK.

So I reverted to v1.28.0. I'm not sure there's any solution to this, other than just remaining on the older version.

[Josverl]

Depends on what functionality your code needs.
There are 2 reduced flash variants available for downloads that may help,

Or pre compiling your code to .mpy (to avoid fragmenting memory by the parser & compiler)
Can be done manually, Deploying via tooling that does this for you (ViperIDE, mpytool), Deploying to ROMFS with Mpremote + moy-cross), or freezing into a custom firmware

Or further optimizing your code.

Without knowing anything about your code it is hard to give you more specific guidance.

[dave0003]

Are the 2 reduced flash variants the 512kiB and 1MiB versions? I have enough flash space. Its RAM that's the issue. Would they help for that?

I did try compiling to .mpy, but that didn't seem to help.

The specific thing that fails is this OTA updater that I use:

https://github.com/RangerDigital/senko

I did add some gc.collect()'s to it, which helped a little, but not enough for v1.29.0. I think its loading the files into memory to compare them.

[Josverl]

Are the 2 reduced flash variants the 512kiB and 1MiB versions? I have enough flash space. Its RAM that's the issue. Would they help for that?

I gave it a quick try with the 1-MB variant, but that does not help.

There are multiple issues and sub-optimal implementations in the senko code, but also after improving/fixing several, the central issue seems to be that the esp8266 just runs out of memory retrieving even a single file for the repo payload = urequests.get(url, headers=self.headers)

So possibly that could be avoided by streaming to a temporary file on the filesystem, rather than putting it in memory.

[dave0003]

There's probably room for improvement. I changed "raw" and "github" urls to point to an HTTP address on my LAN where I keep the micropython code. So that probably helps a little since it doesn't need to do HTTPS or TLS.

[shariltumin]

The test script:

import wifi  # connect to my wifi
import gc, time, sys, requests

urls = ("http://httpbin.org/get", 
        "https://httpbin.org/get",
        "https://micropython.org",
        "https://google.com",
        "https://github.com",
        "https://raw.githubusercontent.com/shariltumin/bit-and-pieces/master/README.md")

print(sys.implementation)
gc.collect()

for url in urls:
   print('----------------------------')
#   print('Mem free:', gc.mem_free())
   print(url)
   try:
      res = requests.get(url, headers={})
      print(res.status_code)
      chunk = res.raw.read(80)
      print(chunk)
   except Exception as e:
      print('ERROR:', e)
   finally:
      res.close()
   gc.collect()
   time.sleep(5)

A run log:

$ mpr mount . run test_http.py
Local directory . is mounted at /remote
('192.168.4.88', '255.255.252.0', '192.168.4.1', '192.168.4.1')
(name='micropython', version=(1, 29, 0, ''), _machine='ESP module with ESP8266', _mpy=9990, _build='ESP8266_GENERIC')
----------------------------
http://httpbin.org/get
200
bytearray(b'{\n  "args": {}, \n  "headers": {\n    "Host": "httpbin.org", \n    "X-Amzn-Trace-Id')
----------------------------
https://httpbin.org/get
200
bytearray(b'{\n  "args": {}, \n  "headers": {\n    "Host": "httpbin.org", \n    "X-Amzn-Trace-Id')
----------------------------
https://micropython.org
TLS buffer overflow, record size: 16448 (+5)
ERROR: (-257, 'RECORD_OVERFLOW')
----------------------------
https://google.com
200
bytearray(b'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="no"')
----------------------------
https://github.com
ERROR: memory allocation failed, allocating 2920 bytes
----------------------------
https://raw.githubusercontent.com/shariltumin/bit-and-pieces/master/README.md
ERROR: -40

It seems that the results depend on the server we were trying to connect to and the TLS handshake enforced by the server.

[dave0003]

Interesting. I modified senko.py slightly, by adding error checking and gc.collect()'s, and also to check files from my LAN over HTTP, so HTTPS and TLS is not needed.

For my purposes, it works on v1.28.0 but not v1.29.0. It fails checking main.py which is 8274 bytes.

Mostly, I just wanted to call attention to the situation in case someone else runs into it. I had initially just upgraded to v1.29.0 while I had the ESP8266 at my desk to troubleshoot another issue, but ended up with more problems. It took me awhile to figure out there were multiple issues and the upgrade contributed to one of them.

[Josverl]

It seems that the results depend on the server we were trying to connect to and the TLS handshake enforced by the server.

Hmm that is interesting, so what eats the memory?

[shariltumin]

The ESP8266 uses axTLS. In Makefile we have:

MICROPY_SSL_AXTLS = 1
AXTLS_DEFS_EXTRA = -Dabort=abort_ -DRT_MAX_PLAIN_LENGTH=1024 -DRT_EXTRA=4096

So during the TLS handshake at most ( and if we have contiguous free heap left ) 5KB will be allocated.

• micropython.org need a standard full-sized TLS record of 16KB, so we got overflow error.
• github.com requires modern cipher suites that axTLS does not support.

The RAM was used-up by the axTLS engine during TLS handshake.

[enkhbold470]

senko compares each file by loading all of it into RAM, and that's what breaks on v1.29. urequests.get(url).text holds the response bytes, then a decoded str copy, then the local file for the comparison. Each of those needs its own contiguous block of heap, so an 8 KB main.py needs roughly three 8 KB blocks at once.

Streaming the file in 512-byte chunks and comparing SHA-1 hashes keeps peak use to one small buffer, whatever the file size:

import requests, hashlib, os

BUF = bytearray(512)
MV = memoryview(BUF)

def file_sha1(path):
    h = hashlib.sha1()
    try:
        with open(path, "rb") as f:
            while n := f.readinto(BUF):
                h.update(MV[:n])
    except OSError:
        return None          # no local copy yet
    return h.digest()

def update(url, path):
    """Download url to path only if it differs. Returns True if replaced."""
    tmp = path + ".new"
    h = hashlib.sha1()
    r = requests.get(url)
    try:
        if r.status_code != 200:
            raise OSError("HTTP %d for %s" % (r.status_code, url))
        with open(tmp, "wb") as f:
            while n := r.raw.readinto(BUF):
                f.write(MV[:n])
                h.update(MV[:n])
    finally:
        r.close()
    if h.digest() == file_sha1(path):
        os.remove(tmp)
        return False
    os.rename(tmp, path)
    return True

hashlib.sha1 and := are both enabled in the ESP8266_GENERIC build, and r.raw in current requests (1.1.1) is a BodyStream whose readinto handles both Content-Length and chunked responses.

Over HTTPS, axTLS also needs about 5 KB of contiguous heap for the handshake, as @shariltumin found. Call gc.collect() right before each requests.get, and import requests early, before the heap fragments. Your LAN-over-HTTP setup avoids that cost entirely.

[dave0003]

Thank you for this. I was able to incorporate this into senko.py and its working much better, even on v1.29.

I did add an "else" at the end:

    if h.digest() == file_sha1(path):
        os.remove(tmp)
        return False
    else:
        os.rename(tmp, path)
        return True

Here's an updated senko.py:

import requests, hashlib, os, machine, time

BUF = bytearray(512)
MV = memoryview(BUF)

class Senko:
    raw = "https://raw.githubusercontent.com"
    github = "https://github.com"

    def __init__(self, user=None, repo=None, url=None, branch=None, working_dir=None, files=["boot.py", "main.py"], headers={}):
        """Senko OTA agent class.

        Args:
            user (str): GitHub user.
            repo (str): GitHub repo to fetch.
            branch (str): GitHub repo branch. (master)
            working_dir (str): Directory inside GitHub repo where the micropython app is.
            url (str): URL to root directory.
            files (list): Files included in OTA update.
            headers (list, optional): Headers for urequests.
        """
        self.base_url = "{}/{}/{}".format(self.raw, user, repo) if user else url.replace(self.github, self.raw)
        self.url = url if url is not None else "{}/{}/{}".format(self.base_url, branch, working_dir)
        self.headers = headers
        self.files = files

    def file_sha1(path):
        h = hashlib.sha1()
        try:
            with open(path, "rb") as f:
                while n := f.readinto(BUF):
                    h.update(MV[:n])
        except OSError:
            return None
        return h.digest()

    def _check_all(self):
        changes = []

        for file in self.files:
            if file is None:
                continue
            url = self.url + "/" + file
            path = file
            """Download url to path only if it differs. Append to changes if replaced."""
            tmp = path + ".new"
            h = hashlib.sha1()
            r = requests.get(url)
            try:
                if r.status_code != 200:
                    raise OSError("HTTP %d for %s" % (r.status_code, url))
                with open(tmp, "wb") as f:
                    while n := r.raw.readinto(BUF):
                        f.write(MV[:n])
                        h.update(MV[:n])
            finally:
                r.close()
            if h.digest() == Senko.file_sha1(path):
                os.remove(tmp)
                print(path + " no change")
            else:
                os.rename(tmp, path)
                changes.append(path)
                print(path + " updated")
        return changes

    def update(self):
        """Replace all changed files with newer one.

        Returns:
            True - if changes were made, False - if not.
        """
        try:
            changes = self._check_all()
            if changes:
                return True
            else:
                return False
        except Exception as e:
            print("senko exception: ", e)
            time.sleep(30)
            machine.reset()

[projectgus]

I think @enkhbold470's reply above is a good overview of this specific problem.

I also want to point out the bigger picture, which is that MicroPython on ESP8266 will only get less viable in future releases. We spend a lot of effort trying to keep code size and memory usage low, but bugfixes tend to add special cases not remove them and as a result code size very slowly creeps up over time.

In addition, our ESP8266 port is based on esp-open-sdk which hasn't been maintained for over eight years. Even the newer version of the same ESP8266 NONOS SDK was deprecated by Espressif in 2019. The axTLS library is also abandonware.

We'll keep the wheels on for as long as we can, for anyone who isn't bothered by being stuck with old TLS and known-vulnerable Wi-Fi.

However, if you are reading this and you are in a position to migrate to any ESP32 chip (ESP32-C3 for example): a lot of things will become easier immediately, and you should have a better time updating to new MicroPython releases for the foreseeable future.

[projectgus]

I've pushed a PR to add some more explicit warnings to the docs for Tier 3 ports like ESP8266, and to try and guide people towards ESP32 where possible: #19862

[dave0003]

Thank you.