Cannot import requests/certifi from embedded zipfile since 2020.4.5.2 release

Just a simple "import requests" no longer works in the context of embedded Python with the code in a zip file. It works fine in 2020.4.5.1.

Traceback (most recent call last):
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\resources.py", line 283, in open_resource
OSError: [Errno 0] Error: 'certifi\\cacert.pem'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "...", line 8, in <module>
    import requests, zeep
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "....\python37.zip\site-packages\requests\__init__.py", line 112, in <module>
    from . import utils
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "....\python37.zip\site-packages\requests\utils.py", line 40, in <module>
    DEFAULT_CA_BUNDLE_PATH = certs.where()
  File "....\python37.zip\site-packages\certifi\core.py", line 37, in where
    _CACERT_PATH = str(_CACERT_CTX.__enter__())
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\contextlib.py", line 112, in __enter__
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\resources.py", line 201, in path
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\resources.py", line 91, in open_binary
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\resources.py", line 285, in open_resource
FileNotFoundError: certifi/cacert.pem

@dstufft

Hey @dstufft , what’s going on here?

How was the zip file created? Does it have the pem file in it as expected?

…

Sent from my iPhone

On Jun 9, 2020, at 2:19 PM, Cory Benfield ***@***.***> wrote: Hey @dstufft , what’s going on here? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

The zip file was created by taking the standard python embeddable zip file and adding the requests module with dependencies to its standard library zip file (python37.zip) for embedded use in a C++ program. It contains the pem file as expected.

I seem to be able to resolve a certificates file from the latest certifi when it's a wheel:

draft $ pip download --no-deps certifi
Collecting certifi
  Using cached certifi-2020.4.5.2-py2.py3-none-any.whl (157 kB)
  Saved ./certifi-2020.4.5.2-py2.py3-none-any.whl
Successfully downloaded certifi
draft $ env PYTHONPATH=certifi-2020.4.5.2-py2.py3-none-any.whl python -c "import certifi, os; print(certifi.where()); print(os.path.isfile(certifi.where()))"
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpl33t7qjo
True

The traceback you show there indicates that the right code is being called. And for whatever reason, self.zipimporter.get_data('certifi/cacert.pem') raises an OSError.

Can you inspect the code at that point and ascertain why that call fails to find that resource?

Alternately, can you suggest a way I could replicate the issue?

Are you able to test easily with Python 3.8? Python 3.8 changed the way zipimport was implemented, so I'd like to eliminate that as a factor if it's not too much trouble.

Finally got round to testing this with Python 3.8.3. Still get a stacktrace, pasted below.

Traceback (most recent call last):
  File "<frozen zipimport>", line 177, in get_data
KeyError: 'certifi\\cacert.pem'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen zipimport>", line 741, in open_resource
  File "<frozen zipimport>", line 179, in get_data
OSError: [Errno 0] : 'certifi\\cacert.pem'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   ...
    import requests, zeep
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File ".....\python38.zip\site-packages\requests\__init__.py", line 112, in <module>
    from . import utils
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File ".....\python38.zip\site-packages\requests\utils.py", line 40, in <module>
    DEFAULT_CA_BUNDLE_PATH = certs.where()
  File "....\python38.zip\site-packages\certifi\core.py", line 37, in where
    _CACERT_PATH = str(_CACERT_CTX.__enter__())
  File "contextlib.py", line 113, in __enter__
  File "importlib\resources.py", line 201, in path
  File "importlib\resources.py", line 91, in open_binary
  File "<frozen zipimport>", line 743, in open_resource
FileNotFoundError: certifi/cacert.pem

As for how to reproduce, I basically just add the requests module to the standard embeddable zipfile.

Download the embeddable zipfile python-3.8.3-embed-amd64.zip
Change to its directory
Do "pip install requests -t packages_tmp"
Call the python script (pasted below) with

py make_embedded_python.py python-3.8.3-embed-amd64.zip packages_tmp TestPython

Try to use the resulting zipfile,

C:\work\python-embedded>py
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, r"C:\work\python-embedded\TestPython\python38.zip\site-packages")
>>> import requests
Traceback (most recent call last):
  File "<frozen zipimport>", line 177, in get_data
KeyError: 'certifi\\cacert.pem'
....

Didn't realised you can't attach files to these issues. Here's the little script "make_embedded_python.py", in any case.

import sys, os, shutil
from zipfile import ZipFile, ZIP_DEFLATED
from glob import glob

def has_pyd_files(path):
    for _, _, files in os.walk(path):
        for fn in files:
            if fn.endswith(".pyd"):
                return True
    return False

def add_directory_to_zip(zf, src, dst):
    for root, _, files in os.walk(src):
        for fn in files:
            srcpath = os.path.join(root, fn)
            srcrelpath = srcpath.replace(src + os.sep, "")
            arcname = os.path.join(dst, srcrelpath)
            zf.write(srcpath, arcname)

if __name__ == "__main__":
    embed_zip = sys.argv[1]
    packages_src = sys.argv[2]
    dest_dir = sys.argv[3]
    if os.path.isdir(dest_dir):
        shutil.rmtree(dest_dir)
    print("  ========= Creating", dest_dir)
    os.makedirs(dest_dir)
        
    with ZipFile(embed_zip, 'r') as zf:
        zf.extractall(dest_dir)

    packages_zip = glob(os.path.join(dest_dir, "python*.zip"))[0]
    with ZipFile(packages_zip, 'a', ZIP_DEFLATED) as zf:
        for fn in os.listdir(packages_src):
            path = os.path.join(packages_src, fn)
            if os.path.isfile(path):
                if fn.endswith(".pyd"):
                    shutil.copy(path, dest_dir)
                else:
                    zf.write(path, os.path.join("site-packages", fn))
            elif not fn.endswith("-info"):
                if has_pyd_files(path):
                    shutil.copytree(path, os.path.join(dest_dir, fn))
                else:
                    add_directory_to_zip(zf, path, os.path.join("site-packages", fn))

I've been able to replicate the issue much more simply:

python -m pip install -t site-packages requests
python -m zipfile -c packages.zip site-packages
python -c "import shutil; shutil.rmtree('site-packages')"
env PYTHONPATH=./packages.zip/site-packages python -c "import requests"

Output

Collecting requests
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Installing collected packages: certifi, idna, chardet, urllib3, requests
Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 urllib3-1.25.9
Traceback (most recent call last):
  File "<frozen zipimport>", line 177, in get_data
KeyError: 'certifi/cacert.pem'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen zipimport>", line 741, in open_resource
  File "<frozen zipimport>", line 179, in get_data
OSError: [Errno 0] : 'certifi/cacert.pem'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/Users/jaraco/draft/packages.zip/site-packages/requests/__init__.py", line 120, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/Users/jaraco/draft/packages.zip/site-packages/requests/utils.py", line 40, in <module>
  File "/Users/jaraco/draft/packages.zip/site-packages/certifi/core.py", line 37, in where
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/resources.py", line 201, in path
    with open_binary(package, resource) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/resources.py", line 91, in open_binary
    return reader.open_resource(resource)
  File "<frozen zipimport>", line 743, in open_resource
FileNotFoundError: certifi/cacert.pem

The issue here appears to be related to the subdirectory on the zipfile (packages.zip/site-packages).

This behavior appears to be a bug in importlib-resources. I'll dig deeper.

I do note that the more modern construct of importlib.resources.files() (new in Python 3.9) doesn't exhibit the issue:

$ env PYTHONPATH=./packages.zip/site-packages pip-run -q importlib_resources -- -c "import importlib_resources; certs = importlib_resources.files('certifi') / 'cacert.pem'; print(len(certs.read_text()))"
282394

I filed python/importlib_resources#105 to track the issue in the package. It may prove difficult to include this behavior in Python 3.8 and impossible to fix in Python 3.7, so my best recommendation is for certifi to use the files API and rely on importlib_resources on Python < 3.9.

I think the bug is also in certifi:

In /certifi/core.py there is a try/except, but the symbol can always be imported. Any implementation that relies on
from certifi import where

will always get the importlib.resources version, never the other version.

So having ONE definition of where(), and a try except inside it would also be a solution.

Switch to more recent 'files' API from importlib.resources from Pytho…

Recent changes to use importlib.resources should resolve this.

Navigation Menu

Repository navigation

Cannot import requests/certifi from embedded zipfile since 2020.4.5.2 release #131

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cannot import requests/certifi from embedded zipfile since 2020.4.5.2 release #131

Description

Activity

Lukasa commented on Jun 9, 2020

dstufft commented on Jun 9, 2020

gjb1002 commented on Jun 10, 2020

jaraco commented on Jun 13, 2020

jaraco commented on Jun 13, 2020

gjb1002 commented on Jun 29, 2020

gjb1002 commented on Jun 29, 2020

gjb1002 commented on Jun 29, 2020

jaraco commented on Jun 29, 2020

jaraco commented on Jun 29, 2020

jaraco commented on Jun 29, 2020

ozio85 commented on Jan 20, 2021

alex commented on Jul 15, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions