* Initial support for SDXL refiner
* Cleanup
* Add arg for converting Unet in float32 precision
* Setup scale factor with pipeline in CLI
* Update cli arg and future warning
* Bundle refiner unet if specified
* Update script for bundled refiner
- Also skip loading model if check_output_correctness is missing, since the model does not require inferencing at conversion time
* Flip skip_model_load bool
* Cleanup
* Support bundled UnetRefiner
* Add seperate refiner config value
- Includes unloading base unet when swapping to refiner
* Update readme for SDXL refiner
* Add condition for new SDXL coreml input features
* Revert pipeline interface change, add extra logging on pipe load
* Reset model_version after refiner conversion
* Reset model_version before refiner conversion but after pipe init
* Add refiner chunking
* Ensure unets are unloaded for reduceMemory true
* Handle missing UnetRefiner.mlmodelc on pipeline load
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Prewarm refiner on load, unload on complete
* Force cpu_and_gpu for VAE until it can be fixed
* Include output dtype of np.float32 for all conversions
* Allow a custom VAE to be converted.
* Revert hardcoded reduceMemory
* Fix merge
* Default chunking arg for --merge-chunks-in-pipeline-model when called from torch2coreml
---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
https://github.com/apple/coremltools/issues/1680 has been resolved in
coremltools 7.0b1, which is now required. The reason for addressing this
change now is that reloading the model before save may incur a
performance cost (specially on SDXL models) due to `ANECompilerService`
triggering and executing for a long time.
* fix multiple issues in mixed bit compression pre analysis: wrong dtype, cannot change default nbits and ImportError
* fixed dtype issue in fake_palettize function
---------
Co-authored-by: Thibault <thibault@Thibaults-Mac-mini.local>
* Extend ControlNet image conversion with hole punching for InPaint
The inpainting ControlNet model expects the masked areas to be filled with -1. This never occured with the original conversion, as the pixel data was converted to values between [0;1], thus the inpainting CN model never had any effect (apart from maybe generating artifacts).
This commit uses the _previously discarded_ alpha channel of the source image to mark masked pixels: any non-opaque pixel is cleared out to -1.
For an end-user, this allows creating masks simply by erasing areas with Preview to transparent, saving the result as PNG, and feeding it to the ControlNet.
* Mask only fully transparent pixels
* Disable SDPA (PyTorch 2.0) in the VAE
Diffusers 0.16.0 made sdpa enabled by default in the vae. It had
previously been enabled for the UNet, but it didn't affect this repo
because it uses its own copy of the UNet.
This fix will not work for diffusers 0.16.0, but it will for main or
0.16.1 (to be released today).
* Link to coremltools issue.
We currently display a warning if the ControlNet repo does not contain a
`base_model` property and the `model-version` is not SD 1.5.
If the ControlNet repo does indicate a `base_model` property, then we
raise a hard error if the versions don't match.
* add controlnet tentatively
* add controlnet in python code
* implement swift part
* support 8-bit quantization
* add controlnet unload when reduce memory
* remove irrelevant changes
* add more description about controlnet option in swift
* fix some for pr and update README
* pre-allocate zero shapedArray + make multi-controlnet faster
Atila Orhon