This will also make ml-stable-diffusion more tolerant with regard to which swift-argument-parser version it needs, resulting in fewer version conflicts in Xcode projects.
* Initial support for SDXL refiner
* Cleanup
* Add arg for converting Unet in float32 precision
* Setup scale factor with pipeline in CLI
* Update cli arg and future warning
* Bundle refiner unet if specified
* Update script for bundled refiner
- Also skip loading model if check_output_correctness is missing, since the model does not require inferencing at conversion time
* Flip skip_model_load bool
* Cleanup
* Support bundled UnetRefiner
* Add seperate refiner config value
- Includes unloading base unet when swapping to refiner
* Update readme for SDXL refiner
* Add condition for new SDXL coreml input features
* Revert pipeline interface change, add extra logging on pipe load
* Reset model_version after refiner conversion
* Reset model_version before refiner conversion but after pipe init
* Add refiner chunking
* Ensure unets are unloaded for reduceMemory true
* Handle missing UnetRefiner.mlmodelc on pipeline load
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Prewarm refiner on load, unload on complete
* Force cpu_and_gpu for VAE until it can be fixed
* Include output dtype of np.float32 for all conversions
* Allow a custom VAE to be converted.
* Revert hardcoded reduceMemory
* Fix merge
* Default chunking arg for --merge-chunks-in-pipeline-model when called from torch2coreml
---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
https://github.com/apple/coremltools/issues/1680 has been resolved in
coremltools 7.0b1, which is now required. The reason for addressing this
change now is that reloading the model before save may incur a
performance cost (specially on SDXL models) due to `ANECompilerService`
triggering and executing for a long time.
* fix multiple issues in mixed bit compression pre analysis: wrong dtype, cannot change default nbits and ImportError
* fixed dtype issue in fake_palettize function
---------
Co-authored-by: Thibault <thibault@Thibaults-Mac-mini.local>
* Extend ControlNet image conversion with hole punching for InPaint
The inpainting ControlNet model expects the masked areas to be filled with -1. This never occured with the original conversion, as the pixel data was converted to values between [0;1], thus the inpainting CN model never had any effect (apart from maybe generating artifacts).
This commit uses the _previously discarded_ alpha channel of the source image to mark masked pixels: any non-opaque pixel is cleared out to -1.
For an end-user, this allows creating masks simply by erasing areas with Preview to transparent, saving the result as PNG, and feeding it to the ControlNet.
* Mask only fully transparent pixels
msiracusa