Running chat app on m2 iPad Pro with mistral 7B unable to generate text

[developer-yl]

Running iOS 18.0 beta 6 on m2 iPad Pro and m1 max Mac is running 14.6.1 but running XCode 16.0 beta 5 (swit-transformers preview 45e5fbb, CompactSlider 1.1.6, Path.swift 1.4.1, swift-argument-parser 1.5.0)

I have downloaded the https://huggingface.co/apple/mistral-coreml mistral 7B coreml model
I am running the chat app in the preview branch, it is able to load the model in the iPad fine but it gives an error when I press the top right play button to generate text with the default "Write a poem about Valencia"

When starting app:

The model is not found at URL: file:///var/mobile/Containers/Data/Application/A12F6050-340D-48D8-973D-851121212F36/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc
No model could be loaded: Error Domain=com.apple.CoreML Code=0 "The model is not found at URL: file:///var/mobile/Containers/Data/Application/A12F6050-340D-48D8-973D-851121212F36/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc" UserInfo={NSLocalizedDescription=The model is not found at URL: file:///var/mobile/Containers/Data/Application/A12F6050-340D-48D8-973D-851121212F36/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc}
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
Failed to send CA Event for app launch measurements for ca_event_type: 1 event_name: com.apple.app_launch_measurement.ExtendedLaunchMetrics

When loading the model:

Compiling model file:///private/var/mobile/Library/Mobile%20Documents/comappleCloudDocs/Downloads/StatefulMistral7BInstructInt4.mlpackage
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
loc("/var/mobile/Containers/Data/Application/A12F6050-340D-48D8-973D-851121212F36/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/22A5338b/AAE089F8CD4E605BA9D6F96BF8073D646BC469AEFEA3D60CB0B2F4F18D462764/1A37B336B2CBAE9955E761EC42C04E24E7FE93A1583CBB9B222612B5C9D2C73D.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode

When pressing the play button:

Error Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0., NSJSONSerializationErrorIndex=0}

[pcuenca]

Hello @developer-yl! You need to set the HUGGING_FACE_HUB_TOKEN environment variable, because the Mistral tokenizer is retrieved from the original repository, which needs authenticated access. This PR improves the error message, we'll get it merged soon.

[developer-yl]

This token needs to created inside profile -> Create new Access Token?
do I have to click all the below permissions?

and added inside the environmental variables in xcode?

or is it through this command on the mac cli?

huggingface-cli login --token hf_KjwGsxQLUR6ydXXXXXXXXX

I've done the above and still get

The model is not found at URL: file:///var/mobile/Containers/Data/Application/2555238C-57D9-42AF-B93E-1E512AF74909/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc
No model could be loaded: Error Domain=com.apple.CoreML Code=0 "The model is not found at URL: file:///var/mobile/Containers/Data/Application/2555238C-57D9-42AF-B93E-1E512AF74909/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc" UserInfo={NSLocalizedDescription=The model is not found at URL: file:///var/mobile/Containers/Data/Application/2555238C-57D9-42AF-B93E-1E512AF74909/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc}
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
Compiling model file:///private/var/mobile/Containers/Shared/AppGroup/3B64D44F-A0A4-4C4D-9218-42511299A15F/File%20Provider%20Storage/Models/StatefulMistral7BInstructInt4%202.mlpackage
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
loc("/var/mobile/Containers/Data/Application/71499946-E4C7-446D-B9FD-56622818114E/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/22A5338b/A1F142919E6DEB3FCCA0BF63EDCD35BC9E8B52C1A262FBE96275EE207A58E80D/DFAC78D6A0B4A8157B3E15396FB8AE12080B94D2EFC069575369FC96B6FF7898.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode
Error Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0., NSJSONSerializationErrorIndex=0}

[pcuenca]

"Read access to gated repos" and setting up in Xcode as in your screenshot should work, as long as you run the app from Xcode.

[developer-yl]

I no longer get the json error but I still get errors does that mean I have proper authenticated access? I did my own conversion from pytorch to coreml as in the preview code in the swift-transformers for export. I have also gained approved of the v0.3 mistral. I have also added the hf_token to the project environmental variables in the schema for debug. Is there any tutorial on how to get the hf_token to work with iOS?

---

Still getting an error attempting to parse a byte at the end of the bytecode:

The model is not found at URL: file:///var/mobile/Containers/Data/Application/71499946-E4C7-446D-B9FD-56622818114E/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4%202.mlmodelc
No model could be loaded: Error Domain=com.apple.CoreML Code=0 "The model is not found at URL: file:///var/mobile/Containers/Data/Application/71499946-E4C7-446D-B9FD-56622818114E/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4%202.mlmodelc" UserInfo={NSLocalizedDescription=The model is not found at URL: file:///var/mobile/Containers/Data/Application/71499946-E4C7-446D-B9FD-56622818114E/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4%202.mlmodelc}
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
Compiling model file:///private/var/mobile/Containers/Shared/AppGroup/3B64D44F-A0A4-4C4D-9218-42511299A15F/File%20Provider%20Storage/Models/StatefulMistral7BInstructInt4.mlpackage
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
loc("/var/mobile/Containers/Data/Application/E7BDDCB3-F0C1-45C1-B8DE-B6DC8CCD8149/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/22A5338b/8A34E0075590CBA0D4E3D17607657A342176E7933CFF51AB5E747B0D5BB2DCD5/A66FCD134CE405370E105AD9E8EFF72697B3085108DF38B66730D674AA5FB6B4.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode

---

I have also attempted to redownload the coreml mistral from apple's page but I still get the same warnings as before with the JSON, so this would likely mean the hf_token isn't working?:

The model is not found at URL: file:///var/mobile/Containers/Data/Application/E7BDDCB3-F0C1-45C1-B8DE-B6DC8CCD8149/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc
No model could be loaded: Error Domain=com.apple.CoreML Code=0 "The model is not found at URL: file:///var/mobile/Containers/Data/Application/E7BDDCB3-F0C1-45C1-B8DE-B6DC8CCD8149/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc" UserInfo={NSLocalizedDescription=The model is not found at URL: file:///var/mobile/Containers/Data/Application/E7BDDCB3-F0C1-45C1-B8DE-B6DC8CCD8149/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc}
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
Compiling model file:///private/var/mobile/Library/Mobile%20Documents/comappleCloudDocs/Downloads/StatefulMistral7BInstructInt4.mlpackage
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
loc("/var/mobile/Containers/Data/Application/33B1DE25-B875-4127-9812-6C196653FA3B/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/22A5338b/D3A6C1807DC917AB651D6ED0957F2CD64FCD98F31C10F71A6E16FF431AAAE9E5/BF13BB7CBBDC3D1385EB7DC3224456443179E590C66C1B2F91E6795AC8BFF5B4.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode
onChange(of: Float) action tried to update multiple times per frame.
Error Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0., NSJSONSerializationErrorIndex=0}

[developer-yl]

"as long as you run the app from Xcode." you mean that it can't run on a device like an ipad?

I have updated the repo with the recently merged transformers preview f4ab454

I now get the message

But I have verified that i have the hugging face token in the project, does the error still relate to a missing hf key?

---

Optional("hf_ROaeclRADxxxxxxxxxxxxxxxxxx")
The model is not found at URL: file:///var/mobile/Containers/Data/Application/B928BC23-8CF0-475A-83DA-2125935A25DF/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc
No model could be loaded: Error Domain=com.apple.CoreML Code=0 "The model is not found at URL: file:///var/mobile/Containers/Data/Application/B928BC23-8CF0-475A-83DA-2125935A25DF/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc" UserInfo={NSLocalizedDescription=The model is not found at URL: file:///var/mobile/Containers/Data/Application/B928BC23-8CF0-475A-83DA-2125935A25DF/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc}
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
Compiling model file:///private/var/mobile/Library/Mobile%20Documents/comappleCloudDocs/Downloads/StatefulMistral7BInstructInt4.mlpackage
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
loc("/var/mobile/Containers/Data/Application/9DB1D7C3-B491-4EE8-AA82-773A8E81F461/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/22A5338b/75DB338E6D3180054D88FA0CDD8DFEC3317C2230E67AFD422CD944F49250DA69/5369FB6AE372BB97191C84BEBDE99791B7B3F4260F0940456C3304589FA3F7DA.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode
Error jsonSerialization(fileURL: file:///var/mobile/Containers/Data/Application/9DB1D7C3-B491-4EE8-AA82-773A8E81F461/Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/config.json, message: "JSON Serialization failed for file:///var/mobile/Containers/Data/Application/9DB1D7C3-B491-4EE8-AA82-773A8E81F461/Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/config.json. Please verify that you have set the HUGGING_FACE_HUB_TOKEN environment variable.")

[developer-yl]

@pcuenca I managed to install macOS 15.0 beta 6 into my mac and was able to get the swift-chat with apple's coreml to work, so that means my implementation seems to be right. However once I run this on my ipad ios 18.0 beta 5, it will not work at all. Was this code tested on a physical iOS device before or only on macOS?

---

Log of it working on mac:

NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
loc("/Users/xxx/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/24A5327a/9951BA009D66E8C17B41D4B1DFFF58FEBD3EF328A1EEC9DE9D83281E95115B13/21CC425DFA3A4B715694D3C3A1201EFC0CFF92DFFBB319A6A0C3F74A37D46137.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode
flock failed to lock list file (/var/folders/vn/t92v266j2xv9c306mtf2bb680000gn/C//com.testmodel/com.apple.metal/32024/libraries.list): errno = 35
flock failed to lock list file (/var/folders/vn/t92v266j2xv9c306mtf2bb680000gn/C//com.testmodel/com.apple.metal/16777235_402/functions.list): errno = 35
Took 1.7980940341949463

---

Failure on the ipad:

The model is not found at URL: file:///var/mobile/Containers/Data/Application/495AC6EB-E56D-4279-B420-0B539B15C197/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc
No model could be loaded: Error Domain=com.apple.CoreML Code=0 "The model is not found at URL: file:///var/mobile/Containers/Data/Application/495AC6EB-E56D-4279-B420-0B539B15C197/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc" UserInfo={NSLocalizedDescription=The model is not found at URL: file:///var/mobile/Containers/Data/Application/495AC6EB-E56D-4279-B420-0B539B15C197/Library/Application%20Support/hf-compiled-transformers/StatefulMistral7BInstructInt4.mlmodelc}
NSBundle file:///System/Library/PrivateFrameworks/MetalTools.framework/ principal class is nil because all fallbacks have failed
-[RTIInputSystemClient remoteTextInputSessionWithID:performInputOperation:] perform input operation requires a valid sessionID. inputModality = Keyboard, inputOperation = , customInfoType = UIEmojiSearchOperations
-[RTIInputSystemClient remoteTextInputSessionWithID:performInputOperation:] perform input operation requires a valid sessionID. inputModality = Keyboard, inputOperation = , customInfoType = UIEmojiSearchOperations
Unable to simultaneously satisfy constraints.
Probably at least one of the constraints in the following list is one you don't want.
Try this:
(1) look at each constraint and try to figure out which you don't expect;
(2) find the code that added the unwanted constraint or constraints and fix it.
(
"<NSLayoutConstraint:0x30243f390 'accessoryView.bottom' _UIRemoteKeyboardPlaceholderView:0x111cc9600.bottom == _UIKBCompatInputView:0x10e7ac380.top (active)>",
"<NSLayoutConstraint:0x30240fb10 'assistantHeight' SystemInputAssistantView.height == 55 (active, names: SystemInputAssistantView:0x104302400 )>",
"<NSLayoutConstraint:0x30243ed50 'assistantView.bottom' SystemInputAssistantView.bottom == _UIKBCompatInputView:0x10e7ac380.top (active, names: SystemInputAssistantView:0x104302400 )>",
"<NSLayoutConstraint:0x30243ff20 'assistantView.top' V:[_UIRemoteKeyboardPlaceholderView:0x111cc9600]-(0)-[SystemInputAssistantView] (active, names: SystemInputAssistantView:0x104302400 )>"
)

Will attempt to recover by breaking constraint
<NSLayoutConstraint:0x30240fb10 'assistantHeight' SystemInputAssistantView.height == 55 (active, names: SystemInputAssistantView:0x104302400 )>

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
Compiling model file:///private/var/mobile/Library/Mobile%20Documents/comappleCloudDocs/Downloads/StatefulMistral7BInstructInt4.mlpackage
The view service did terminate with error: Error Domain=_UIViewServiceErrorDomain Code=1 "(null)" UserInfo={Terminated=disconnect method}
loc("/var/mobile/Containers/Data/Application/A53DD8B1-0377-40D6-829D-9D91B6A32665/Library/Caches/com.testmodel/com.apple.e5rt.e5bundlecache/22A5338b/9959EAAB576A4626E6EE869C3BA1FBFCADD5B9E69C8D39B9533C43D374ABB16E/B7D1254526122D28B82F274D324D26891089EF5283E22F11F4298447095AC2B4.bundle/H14G.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode
Error jsonSerialization(fileURL: file:///var/mobile/Containers/Data/Application/A53DD8B1-0377-40D6-829D-9D91B6A32665/Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/config.json, message: "JSON Serialization failed for file:///var/mobile/Containers/Data/Application/A53DD8B1-0377-40D6-829D-9D91B6A32665/Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/config.json. Please verify that you have set the HUGGING_FACE_HUB_TOKEN environment variable.")
Message from debugger: killed

[pixelcatgg]

+1 on this, i've set the variable still not able to get past the "Please verify that you have set the HUGGING_FACE_HUB_TOKEN environment variable"

[pixelcatgg]

@developer-yl Grateful if you can share how you got apple's core-ml version to work on Mac OS. i am downloading it like this per the huggingface readme:

huggingface-cli download \
  --local-dir models \
  --local-dir-use-symlinks False \
  apple/mistral-coreml \
  --include "StatefulMistral7BInstructInt4.mlpackage/*"

I have set the token and validated with a print statement that it is there.

I am on MacOS 15 beta

Building for Mac.

Cannot get past the token issue! its killing me

[pixelcatgg]

Update I got it working, i had to manually set the token into my MacOS CLI (outside of Xcode)

export HUGGING_FACE_HUB_TOKEN=hf_example_token not into XCode.

You can verify it works because it creates a folder under Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/config.json. If you are unathenticated the contents of that file will show an error message telling you so, but if it works it should look like this

  "architectures": [
    "MistralForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 32768,
  "model_type": "mistral",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "rms_norm_eps": 1e-05,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.42.0.dev0",
  "use_cache": true,
  "vocab_size": 32768
}

My hunch is that on a device like iPad this will not work that way. I wonder if there's a way to load the tokenizer functions into the device manually or something?

[sgallese]

Setting the Environment Variable in Xcode
If you still have trouble setting the environment variable on your system, you can set Environment Variables in Xcode directly.

In Xcode, if you tap SwiftChat -> Edit Scheme, you can enter in HUGGING_FACE_HUB_TOKEN under Environment Variables with your key.

Agree to Mistral access
If you verify your token works and you still receive the Please verify that you have set the HUGGING_FACE_HUB_TOKEN environment variable. message, check /Users/<YOUR_USER_NAME>/Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/config.json

I found the file had an error not related to environmental variable:
Access to model mistralai/Mistral-7B-Instruct-v0.3 is restricted and you are not in the authorized list. Visit https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 to ask for access.

So I went to https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 and clicked "Agree and access repository"

Clear Json cache
Lastly, you may still see the same error after doing all of the above!

That's because the .json files are cached at /Users/<YOUR_USER_NAME>/Documents/huggingface/models/mistralai/Mistral-7B-Instruct-v0.3/.

Make sure to delete all three .json files and restart SwiftChat!

I spent 20 mins thinking my token fine-grained permission was wrong, but all you need is Read access to contents of all public gated repos you can access and to clear those .json files once you agree to Mistral access above.