Different versions of python between debug and notarized versions

[ajram23]

Describe the bug

I used Python-3.11-macOS-support.b1.tar.gz and imported both python-stdlib and Python.xcframework. Am I missing something basic?

Steps to reproduce

Here is what the version file says Python version: 3.11.0
Build: b1
Min macOS version: 10.15

libFFI: macOS native
BZip2: 1.0.8
OpenSSL: 3.0.5
XZ: 5.2.6
Here is my swift code: import SwiftUI
import Python
import PythonKit
import Foundation

struct PythonInfo {
var version: String = ""
var encoding: String = ""
var path: String = ""
}

class EmbeddedPython: ObservableObject {
@ Published var pythonInfo = PythonInfo()
private let logFile: URL

init() {
let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
let appFolder = appSupport.appendingPathComponent("PythonSwiftoo", isDirectory: true)
try? FileManager.default.createDirectory(at: appFolder, withIntermediateDirectories: true)
logFile = appFolder.appendingPathComponent("python_log.txt")

logMessage("EmbeddedPython initialization started")
logMessage("Current working directory: \(FileManager.default.currentDirectoryPath)")

// Log environment variables
for (key, value) in ProcessInfo.processInfo.environment {
    logMessage("Environment variable: \(key) = \(value)")
}

if let stdLibPath = Bundle.main.path(forResource: "python-stdlib", ofType: nil),
   let libDynloadPath = Bundle.main.path(forResource: "python-stdlib/lib-dynload", ofType: nil)
{
    logMessage("Found stdlib path: \(stdLibPath)")
    logMessage("Found lib-dynload path: \(libDynloadPath)")
    
    // Log contents of lib-dynload directory
    if let contents = try? FileManager.default.contentsOfDirectory(atPath: libDynloadPath) {
        logMessage("Contents of lib-dynload: \(contents)")
    }
    
    setenv("PYTHONHOME", stdLibPath, 1)
    setenv("PYTHONPATH", "\(stdLibPath):\(libDynloadPath)", 1)
    logMessage("Environment variables set")
    logMessage("PYTHONHOME set to: \(String(cString: getenv("PYTHONHOME")))")
    logMessage("PYTHONPATH set to: \(String(cString: getenv("PYTHONPATH")))")
    
    logMessage("Before Py_Initialize()")
    Py_Initialize()
    logMessage("After Py_Initialize()")
} else {
    logMessage("Failed to find Python paths")
}

updatePythonInfo()

}

func updatePythonInfo() {
logMessage("Updating Python info")

do {
    let sys = try Python.import("sys")
    logMessage("sys module imported")
    
    self.pythonInfo.version = "\(sys.version_info.major).\(sys.version_info.minor)"
    self.pythonInfo.encoding = "\(sys.getdefaultencoding().upper())"
    self.pythonInfo.path = "\(sys.path)"
    
    logMessage("Python version: \(self.pythonInfo.version)")
    logMessage("Python encoding: \(self.pythonInfo.encoding)")
    logMessage("Python path: \(self.pythonInfo.path)")
    //logMessage("sys.prefix: \(sys.prefix)")
    logMessage("sys.exec_prefix: \(sys.exec_prefix)")
    logMessage("sys.executable: \(sys.executable)")
    
    let os = try Python.import("os")
    logMessage("Current working directory (from Python): \(os.getcwd())")
    
    _ = try Python.import("math")
    logMessage("math module imported successfully")
} catch {
    logMessage("Error in updatePythonInfo: \(error)")
}

}

func logMessage(_ message: String) {
let timestamp = ISO8601DateFormatter().string(from: Date())
let logEntry = "[(timestamp)] (message)\n"

do {
    if FileManager.default.fileExists(atPath: logFile.path) {
        let fileHandle = try FileHandle(forWritingTo: logFile)
        fileHandle.seekToEndOfFile()
        fileHandle.write(logEntry.data(using: .utf8)!)
        fileHandle.closeFile()
    } else {
        try logEntry.write(to: logFile, atomically: true, encoding: .utf8)
    }
} catch {
    print("Failed to write to log file: \(error.localizedDescription)")
}

}
}

@ main
struct PythonSwiftooApp: App {
@ StateObject var embeddedPython = EmbeddedPython()

var body: some Scene {
WindowGroup {
ContentView(embeddedPython: embeddedPython)
.onAppear {
embeddedPython.logMessage("App appeared")
}
}
}
} and yet if you look at the logs it says python version 3.12 and the app crashes after it launches [2024-09-05T22:38:16Z] EmbeddedPython initialization started

[2024-09-05T22:38:16Z] Current working directory: /

[2024-09-05T22:38:16Z] Environment variable: XPC_FLAGS = 0x0

[2024-09-05T22:38:16Z] Environment variable: TMPDIR = /var/folders/bc/hwycx8wx3151dwgk0nkwv2c00000gn/T/

[2024-09-05T22:38:16Z] Environment variable: HOME = /Users/ajr

[2024-09-05T22:38:16Z] Environment variable: PATH = /usr/bin:/bin:/usr/sbin:/sbin

[2024-09-05T22:38:16Z] Environment variable: USER = ajr

[2024-09-05T22:38:16Z] Environment variable: SHELL = /bin/zsh

[2024-09-05T22:38:16Z] Environment variable: COMMAND_MODE = unix2003

[2024-09-05T22:38:16Z] Environment variable: LOGNAME = ajr

[2024-09-05T22:38:16Z] Environment variable: XPC_SERVICE_NAME = application.com.rolodexguru.PythonSwiftoo.2723382.2723388

[2024-09-05T22:38:16Z] Environment variable: __CFBundleIdentifier = com.rolodexguru.PythonSwiftoo

[2024-09-05T22:38:16Z] Environment variable: SSH_AUTH_SOCK = /private/tmp/com.apple.launchd.CMfn38d7Yi/Listeners

[2024-09-05T22:38:16Z] Environment variable: __CF_USER_TEXT_ENCODING = 0x1F5:0x0:0x0

[2024-09-05T22:38:16Z] Found stdlib path: /Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib

[2024-09-05T22:38:16Z] Found lib-dynload path: /Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/lib-dynload

[2024-09-05T22:38:16Z] Contents of lib-dynload: ["_sqlite3.cpython-311-darwin.so", "_scproxy.cpython-311-darwin.so", "_md5.cpython-311-darwin.so", "_multiprocessing.cpython-311-darwin.so", "_heapq.cpython-311-darwin.so", "_codecs_kr.cpython-311-darwin.so", "_opcode.cpython-311-darwin.so", "_codecs_tw.cpython-311-darwin.so", "_uuid.cpython-311-darwin.so", "_elementtree.cpython-311-darwin.so", "cmath.cpython-311-darwin.so", "_crypt.cpython-311-darwin.so", "_lsprof.cpython-311-darwin.so", "resource.cpython-311-darwin.so", "_codecs_hk.cpython-311-darwin.so", "audioop.cpython-311-darwin.so", "_contextvars.cpython-311-darwin.so", "_random.cpython-311-darwin.so", "_json.cpython-311-darwin.so", "binascii.cpython-311-darwin.so", "_datetime.cpython-311-darwin.so", "mmap.cpython-311-darwin.so", "nis.cpython-311-darwin.so", "_bz2.cpython-311-darwin.so", "_codecs_jp.cpython-311-darwin.so", "_ctypes.cpython-311-darwin.so", "math.cpython-311-darwin.so", "_codecs_iso2022.cpython-311-darwin.so", "_curses.cpython-311-darwin.so", "_dbm.cpython-311-darwin.so", "_ctypes_test.cpython-311-darwin.so", "unicodedata.cpython-311-darwin.so", "termios.cpython-311-darwin.so", "zlib.cpython-311-darwin.so", "_sha256.cpython-311-darwin.so", "_bisect.cpython-311-darwin.so", "_asyncio.cpython-311-darwin.so", "_codecs_cn.cpython-311-darwin.so", "_multibytecodec.cpython-311-darwin.so", "_sha512.cpython-311-darwin.so", "_posixsubprocess.cpython-311-darwin.so", "_pickle.cpython-311-darwin.so", "pyexpat.cpython-311-darwin.so", "select.cpython-311-darwin.so", "_lzma.cpython-311-darwin.so", "readline.cpython-311-darwin.so", "_typing.cpython-311-darwin.so", "syslog.cpython-311-darwin.so", "_blake2.cpython-311-darwin.so", "_queue.cpython-311-darwin.so", "array.cpython-311-darwin.so", "_struct.cpython-311-darwin.so", "_zoneinfo.cpython-311-darwin.so", "_posixshmem.cpython-311-darwin.so", "_sha1.cpython-311-darwin.so", "_decimal.cpython-311-darwin.so", "_socket.cpython-311-darwin.so", "_hashlib.cpython-311-darwin.so", "_curses_panel.cpython-311-darwin.so", "grp.cpython-311-darwin.so", "_ssl.cpython-311-darwin.so", "_sha3.cpython-311-darwin.so", "fcntl.cpython-311-darwin.so", "_statistics.cpython-311-darwin.so", "_csv.cpython-311-darwin.so"]

[2024-09-05T22:38:16Z] Environment variables set

[2024-09-05T22:38:16Z] PYTHONHOME set to: /Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib

[2024-09-05T22:38:16Z] PYTHONPATH set to: /Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib:/Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/lib-dynload

[2024-09-05T22:38:16Z] Before Py_Initialize()

[2024-09-05T22:38:16Z] After Py_Initialize()

[2024-09-05T22:38:16Z] Updating Python info

[2024-09-05T22:38:16Z] sys module imported

[2024-09-05T22:38:16Z] Python version: 3.12

[2024-09-05T22:38:16Z] Python encoding: UTF-8

[2024-09-05T22:38:16Z] Python path: ['/Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib', '/Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/lib-dynload', '/Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/lib/python312.zip', '/Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/lib/python3.12', '/Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/lib/python3.12/lib-dynload']

[2024-09-05T22:38:16Z] sys.exec_prefix: /Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib

[2024-09-05T22:38:16Z] sys.executable: /Users/ajr/Desktop/PythonSwiftoo_More_logs.app/Contents/Resources/python-stdlib/bin/python3.12

[2024-09-05T22:38:16Z] Current working directory (from Python): / .

Expected behavior

Same version of python

Screenshots

No response

Environment

Python version: 3.11.0
Build: b1
Min macOS version: 10.15

libFFI: macOS native
BZip2: 1.0.8
OpenSSL: 3.0.5
XZ: 5.2.6

Logs

see above

Additional context

No response

[freakboy3742]

I don't know how you've gotten into this situation, but you've clearly downloaded a version of Python 3.12 at some point. The log indicates that it is loading python312.zip (amongst other files), which is a file that definitely isn't in the 3.11-b1 support package you've referenced.

If you're doing this on your own Swift macOS app, it's likely that you're picking up the system Python (i.e., the one installed in /Library/Frameworks or in your homebrew /opt folder) at runtime.

Closing on the basis that there's no evidence this a bug in the support package. Use of the support package outside Briefcase isn't a configuration we actively support; we're open to improving our documentation in this regard, but for now at least, a stub Briefcase project is our documentation of "how it should work".

[ajram23]

@freakboy3742 Thank you for the prompt response, I see that now. This seems to be happening with my latest code as well and I can't seem to find any python312.zip. Will find other ways.

When you say " Use of the support package outside Briefcase isn't a configuration we actively support" I am assuming swift files that invoke through pythonKit fall under this use case.

Lastly is there any documentation that you can point me towards threading issues or constraints? Surprisingly I have it all running mostly barring the python 3.12 & a threading issue and hence the question. Regardless appreciate all the work for this library, it's a godsend for folks developing local AI apps in python! Thank you and I am grateful for it.

[freakboy3742]

@freakboy3742 Thank you for the prompt response, I see that now. This seems to be happening with my latest code as well and I can't seem to find any python312.zip. Will find other ways.

When you say " Use of the support package outside Briefcase isn't a configuration we actively support" I am assuming swift files that invoke through pythonKit fall under this use case.

Broadly speaking yes. I'm happy to entertain improvements to the documentation that make it easier for others to use PythonKit or manually embed this support package, or patches that might be needed to improve PythonKit integration; but our primary focus isn't the PythonKit use case - it's Briefcase.

Lastly is there any documentation that you can point me towards threading issues or constraints? Surprisingly I have it all running mostly barring the python 3.12 & a threading issue and hence the question.

In terms of basic operation, I'm not aware of any limitations. It's the same CPython that runs anywhere else on macOS. Depending on exactly how you've embedded the library, there might be issues with multiprocessing, but threading shouldn't have any limitations that don't already exist with "stock" CPython.

[ajram23]

@freakboy3742 Thank you for the prompt response, I see that now. This seems to be happening with my latest code as well and I can't seem to find any python312.zip. Will find other ways.
When you say " Use of the support package outside Briefcase isn't a configuration we actively support" I am assuming swift files that invoke through pythonKit fall under this use case.

Broadly speaking yes. I'm happy to entertain improvements to the documentation that make it easier for others to use PythonKit or manually embed this support package, or patches that might be needed to improve PythonKit integration; but our primary focus isn't the PythonKit use case - it's Briefcase.

Understandable!

Lastly is there any documentation that you can point me towards threading issues or constraints? Surprisingly I have it all running mostly barring the python 3.12 & a threading issue and hence the question.

In terms of basic operation, I'm not aware of any limitations. It's the same CPython that runs anywhere else on macOS. Depending on exactly how you've embedded the library, there might be issues with multiprocessing, but threading shouldn't have any limitations that don't already exist with "stock" CPython.
The issue seems to be with threads and specifically where it's invoked from and hence the question. I am invoking them as a python algo.py service Also I am assuming that invoking two python scripts in my case running as flask will be problematic?
Lastly any known issues with notarization? The embedded python is not being used in a notarized exec, not sure if there's something basic I am missing. I have already disabled the hardened environment & disabled library validation in the signing & capabilities. Thanks again for the help!

[ajram23]

@freakboy3742 Did something change from releases between python-311-b3 and b4? You went from including std-lib and not have python exec to not including std-lib in your releases but have a new python exec and other libs. while I know this is a little older information. Unfortunately this seems to best doc out there for PythonKit and your library integration. https://butternut-ray-869.notion.site/Embedding-Python-in-a-Swift-App-Using-Xcode-Apple-Silicon-and-Intel-59f5a3c832914a96990dfcc0e610c720

[freakboy3742]

Yes. As noted in the release for 3.11-b4, we moved to a new frameworkpackage structure that is consistent with the patches that were used to implement PEP 730.

[ajram23]

@freakboy3742 aah ok thanks! Curious what happens to python-stdlib? Cuz import Python no longer works. Or is that someone else's problem? Just trying to figure out how to get this running with your latest releases. Apologies if I am being persistent. You have been very generous!

[freakboy3742]

@freakboy3742 aah ok thanks! Curious what happens to python-stdlib? Cuz import Python no longer works.

It's now distributed as a lib folder inside the appropriate "slice" of the XCframework (so - Python.xcframeworks/ios-arm64/lib. This means that the "slice" can also act as PYTHONHOME for the purposes of configuration.