Skip to content

Commit

Permalink
Add Core ML functionality (#766)
Browse files Browse the repository at this point in the history
* test commit

* overload classify function with coreml version. hit standard VR endpoint on callback

* iterate coreml results on callback

* convert coreml output into vision SDK model format

* filter coreml results by confidence. return immediately if valid results

* don't do coreml request on background thread

* add debug line...

* cleaned up printing, return more responses

* convert coreml classify function to accept image as NSData

* run classification on background thread to be non-blocking

* test passing to sdk as NSdata

* test

* classify function for NSData by writing to temp file... giving up on proper implementation for now

* short-circuit coreml call if localthresh min is set to 1.0

* moving model abstraction into sdk

* Completion handler for getLatest method

* moved coreml functions to extension of VR class

* switch from passing image as NSData to Data

* fix: JSON -> JSONWrapper

* point at bluemix server for model pull. this is only temporary until coreml functionality is moved to VR server. this requires keys to be placed at Source/VisualRecognitionV3/Credentials.swift

* pass api key via method

* update getClassifiers with  and updated classifier data model

* updated createClassifier with core_ml_enabled

* fixed some indentation

* more indentation

* typos and refactored core_ml_enabled and core_ml_status to camelCase

* change core_ml_enabled to coreMLEnabled

* updated updateClassifier with coreMLEnabled

* coreMLEnalbed: Bool? = nil

* coreMLEnalbed: Bool? = nil

* coreMLEnabled defaults to nil, check param before adding

* check if coreMLEnabled has value, stringify bool

* check if coreMLEnabled has value, stringify bool

* removed tab

* add space

* downloadClassifier modeled after downloading configuration in R&R

* revised endpoint for downloadClassifier

* update documentation annotation

* unwrap hybrid classify function

* decompose model abstraction class. store models by classifier id. first versions of updateCoreMLModelLocally and getCoreMLModelLocally

* store models in application support directory

* success and failure for callbacks to updateLocalCoreMLModel

* handle requests to multiple local classifiers. something wrong around variable scoping and callback closures

* moved downloadClassifier to VR extension, changes from pairing with Glenn

* explicitly classify on different threads.  VNImageRequestHandler was not behaving for multiple requests. this fixes the issue noted in the last commit. also using a dispatch group now

* return MLModel from getCoreMLModeLocally

* parse metadata from model for classifier ID and name

* nicer formatting for unwrapping metadata dict

* exclude saved model file from backup

* changes from glenn's pr review

* Construct compiledModelURL before it is referenced

* Combine do-try-catch blocks for readability

* Use NSLocalizedDescriptionKey instead of NSLocalizedFailureReasonErrorKey

* Check if compiled model exists before removing it

* add downloadClassifier

* orig cleanup

* Extend availability tag for multiple platforms

* Change image type to UIImage

* Update style

* revise updateCoreMLModelLocally to check for new classifier and call download

* Change `results` type to an array to avoid hashing

* Fix errors with JSON representation of results

* Use modelURL.path instead of modelURL.absoluteString

* getCoreMLModelLocally, updateCoreMLModelLocally and downloadClassifier all return URL

* WIP - listCoreMLModels, changed dateformatter

* throw error in listCoreMLmodels, implement deleteCoreMLModel, write getApplicationSupportDirectory

* pairing with Glenn on listCoreMLModels and deleteCoreMLModel

* Add getCoreMLModel

* Set the url for testing

* Fix string interpolation value

* Return MLModel directly instead of URL

* Check the classifier’s core ml status when using preferRemote policy

A newer version of the model is available when (1) the classifier has a newer updated date and (2) the classifier’s core ml status is “ready”.

If the classifier has been updated with new training data but the core ml model is still being trained, then the preferRemote policy will use the local model.

* Use test server for getClassifier

* update ReadMe with Core ML Exampeles

* Use classifier id when contacting test server

* Use a separate api key for the test server

* Download a classifier if the model’s metadata cannot be read

* change header from public api to core ML, regeregeneratte documentation

* move core ml readme above links

* hotfix: hit failure function on all error returns

* refactor from discussed changes

* orig cleanup

* orig cleanup

* revise header from Public API to Core ML

* regenerate documentation

* regenerate documentation

* revise method in readme according to sdk changes

* Filter observations with user-provided threshold value

* Add comments

* Simplify the execution of each classification request

* Revert "Simplify the execution of each classification request"

This reverts commit 4377b70.

* Update comments for test server networking

* Fix url for test server

* Run convert and success with userInitiated qos

* Move scope of defer statement

* Create downloads directory if it doesn’t exist

* Use explicit date formatter with milliseconds

* regenerate documentation

* Break up sentance in readme

* Create application support directory if it does not already exist

* Specify that model files are not directories

* Use replaceItemAt when the compiled model already exists

* revised examples

* rephrased Using Core ML description

* Change interface for getLocalModel

* Add tests for VisualRecognition+CoreML

* regenerate documentation

* rename VisualRecognitionCoreML.swift to VisualRecognition+CoreML.swift

* Do not require the application support directory to exist

* Simplify model lookup in main bundle

* Construct strings with error.localizedDescription

* Improve comments

* Avoid silencing errors that are thrown

* resolving merge conflicts

* leave dispatch group on failure to execute classification request

* Use https instead of http

* getting last updated date, adding retrained to the classifier model

* misspelling, nullable retrained in datamodel, ?? operator to get date

* remove core_ml_status from classifier model - use status

* remove core_ml_enabled query param

* revise documentation for GET/classifiers/{cid} to include IBM Model ID

* add owners query param to getClassifiers operation

* parse local model updated date from created & retrained

* remove updated field from Classifier data model

* create temporary directory for mlmodel download instead of using Documents directory

* regenerate documentationgit status

* refactored out temporary service

* adding missing query params in download core ml model and get classifier, remove coreMLEnabled from classifier

* indentation

* add in commented out coreMLEnabled field to be uncommented when it is available in the response

* change acceptType of DownloadClassifier to application/octet-stream, uncomment coreMlEnabled in model and made nullable

* changing readme ios-sdk references to swift-sdk (mostly in URLs) per legal request

* Make properties internal instead of private

* Update CoreML extension for updated API

* Disable function_body_length rule for downloadClassifier

* Update tests for latest Visual Recognition API

* Update style to conform to SwiftLint

* Create an internal JSON string instead of a dictionary

* Remove extraneous test now that the service is live

* update readme to include separate classify and update instructions, getLocalModel and deleteLocalModel

* implement Glenn's suggestions

* missing 'when'

* add Devin's readme changes

* update readme with glenns suggestion

* devin's suggestions

* Regenerate swift-api documentation

* revise list models method to check if application support directory exists.
  • Loading branch information
mediumTaj authored and glennrfisher committed Mar 20, 2018
1 parent 0b281d1 commit 0b38ed4
Show file tree
Hide file tree
Showing 58 changed files with 2,905 additions and 649 deletions.
56 changes: 52 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
![](https://img.shields.io/badge/platform-iOS,%20Linux-blue.svg?style=flat)
[![Carthage Compatible](https://img.shields.io/badge/Carthage-compatible-4BC51D.svg?style=flat)](https://github.com/Carthage/Carthage)
[![Documentation](https://img.shields.io/badge/Documentation-API-blue.svg)](http://watson-developer-cloud.github.io/swift-sdk)
[![CLA assistant](https://cla-assistant.io/readme/badge/watson-developer-cloud/ios-sdk)](https://cla-assistant.io/watson-developer-cloud/swift-sdk)
[![CLA assistant](https://cla-assistant.io/readme/badge/watson-developer-cloud/swift-sdk)](https://cla-assistant.io/watson-developer-cloud/swift-sdk)

## Overview

Expand Down Expand Up @@ -181,12 +181,12 @@ To use the Watson SDK in your Linux project, please follow the [Swift Package Ma

## Contributing

We would love any and all help! If you would like to contribute, please read our [CONTRIBUTING](https://github.com/watson-developer-cloud/ios-sdk/blob/master/.github/CONTRIBUTING.md) documentation with information on getting started.
We would love any and all help! If you would like to contribute, please read our [CONTRIBUTING](https://github.com/watson-developer-cloud/swift-sdk/blob/master/.github/CONTRIBUTING.md) documentation with information on getting started.

## License

This library is licensed under Apache 2.0. Full license text is
available in [LICENSE](https://github.com/watson-developer-cloud/ios-sdk/blob/master/LICENSE).
available in [LICENSE](https://github.com/watson-developer-cloud/swift-sdk/blob/master/LICENSE).

This SDK is intended for use with an Apple iOS product and intended to be used in conjunction with officially licensed Apple development tools.

Expand Down Expand Up @@ -555,7 +555,7 @@ var settings = RecognitionSettings(contentType: .wav)
settings.interimResults = true
```

See the [class documentation](http://watson-developer-cloud.github.io/ios-sdk/services/SpeechToTextV1/Structs/RecognitionSettings.html) or [service documentation](https://console.bluemix.net/docs/services/speech-to-text/index.html) for more information about the available settings.
See the [class documentation](http://watson-developer-cloud.github.io/swift-sdk/services/SpeechToTextV1/Structs/RecognitionSettings.html) or [service documentation](https://console.bluemix.net/docs/services/speech-to-text/index.html) for more information about the available settings.

#### Microphone Audio and Compression

Expand Down Expand Up @@ -930,6 +930,54 @@ visualRecognition.classify(image: url, failure: failure) { classifiedImages in
}
```

### Using Core ML

The Watson Swift SDK supports offline image classification using Apple Core ML. Classifiers must be trained or updated with the `coreMLEnabled` flag set to true. Once the classifier's `coreMLStatus` is `ready` then a Core ML model is available to download and use for offline classification.

Once the Core ML model is in the device's file system, images can be classified offline, directly on the device.

```swift
let classifierID = "your-classifier-id"
let failure = { (error: Error) in print(error) }
let image = UIImage(named: "your-image-filename")
visualRecognition.classifyWithLocalModel(image: image, classifierIDs: [classifierID], failure: failure) {
classifiedImages in print(classifiedImages)
}
```

The local Core ML model can be updated as needed.

```swift
let classifierID = "your-classifier-id"
let failure = { (error: Error) in print(error) }
visualRecognition.updateLocalModel(classifierID: classifierID, failure: failure) {
print("model updated")
}
```

The following example demonstrates how to list the Core ML models that are stored in the filesystem and available for offline use:

```swift
let localModels = try! visualRecognition.listLocalModels()
print(localModels)
```

If you would prefer to bypass `classifyWithLocalModel` and construct your own Core ML classification request, then you can retrieve a Core ML model from the local filesystem with the following example.
```swift
let classifierID = "your-classifier-id"
let localModel = try! visualRecognition.getLocalModel(classifierID: classifierID)
print(localModel)
```

The following example demonstrates how to delete a local Core ML model from the filesystem. This saves space when the model is no longer needed.
```swift
let classifierID = "your-classifier-id"
visualRecognition.deleteLocalModel(classifierID: classifierID)
```

#### Bundling a model directly with your application
You may also choose to include a Core ML model with your application, enabling images to be classified offline without having to download a model first. To include a model, add it to your application bundle following the naming convention [classifier_id].mlmodel. This will enable the SDK to locate the model when using any function that accepts a classifierID argument.

The following links provide more information about the IBM Watson Visual Recognition service:

* [IBM Watson Visual Recognition - Service Page](https://www.ibm.com/watson/services/visual-recognition/)
Expand Down
12 changes: 10 additions & 2 deletions Source/VisualRecognitionV3/Models/Classifier.swift
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ public struct Classifier {
/// Date and time in Coordinated Universal Time that the classifier was updated. Returned when verbose=`true`. Might not be returned by some requests.
public var retrained: String?

/// If the classifier is CoreML Enabled.
public let coreMLEnabled: Bool?

/**
Initialize a `Classifier` with member variables.

Expand All @@ -62,10 +65,11 @@ public struct Classifier {
- parameter created: Date and time in Coordinated Universal Time that the classifier was created.
- parameter classes: Array of classes that define a classifier.
- parameter retrained: Date and time in Coordinated Universal Time that the classifier was updated. Returned when verbose=`true`. Might not be returned by some requests.
- parameter coreMLEnabled: If the classifier is CoreML Enabled.

- returns: An initialized `Classifier`.
*/
public init(classifierID: String, name: String, owner: String? = nil, status: String? = nil, explanation: String? = nil, created: String? = nil, classes: [Class]? = nil, retrained: String? = nil) {
public init(classifierID: String, name: String, owner: String? = nil, status: String? = nil, explanation: String? = nil, created: String? = nil, classes: [Class]? = nil, retrained: String? = nil, coreMLEnabled: Bool? = nil) {
self.classifierID = classifierID
self.name = name
self.owner = owner
Expand All @@ -74,6 +78,7 @@ public struct Classifier {
self.created = created
self.classes = classes
self.retrained = retrained
self.coreMLEnabled = coreMLEnabled
}
}

Expand All @@ -88,7 +93,8 @@ extension Classifier: Codable {
case created = "created"
case classes = "classes"
case retrained = "retrained"
static let allValues = [classifierID, name, owner, status, explanation, created, classes, retrained]
case coreMLEnabled = "core_ml_enabled"
static let allValues = [classifierID, name, owner, status, explanation, created, classes, retrained, coreMLEnabled]
}

public init(from decoder: Decoder) throws {
Expand All @@ -101,6 +107,7 @@ extension Classifier: Codable {
created = try container.decodeIfPresent(String.self, forKey: .created)
classes = try container.decodeIfPresent([Class].self, forKey: .classes)
retrained = try container.decodeIfPresent(String.self, forKey: .retrained)
coreMLEnabled = try container.decodeIfPresent(Bool.self, forKey: .coreMLEnabled)
}

public func encode(to encoder: Encoder) throws {
Expand All @@ -113,6 +120,7 @@ extension Classifier: Codable {
try container.encodeIfPresent(created, forKey: .created)
try container.encodeIfPresent(classes, forKey: .classes)
try container.encodeIfPresent(retrained, forKey: .retrained)
try container.encodeIfPresent(coreMLEnabled, forKey: .coreMLEnabled)
}

}
Loading

0 comments on commit 0b38ed4

Please sign in to comment.