Media #WWDC17

Advances in Core Image

•

Filters, Metal, Vision, and More

•

Session 510

David Hayward, Core Image Manager

© 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
 Agenda

A brief overview of Core Image

•

Highlights of what’s new this year

•

Details on new APIs and features

A Brief Overview of Core Image
•
A Very Brief Introduction to Core Image
A simple, high-performance API to apply filters to images

Input to Sepia Hue Contrast Working Space

O
Filter Filter Filter to Output

Original CIImage Output CIImage

A Very Brief Introduction to Core Image
Automatically tiles if images are large or graph is complex

Input to Sepia Hue Contrast Working Space

O
Filter Filter Filter to Output

Original CIImage Output CIImage

A Very Brief Introduction to Core Image
Automatically tiles if only a region of the output is rendered

Input to Sepia Hue Contrast Working Space

O
Filter Filter Filter to Output

Original CIImage Output CIImage

A Very Brief Introduction to Core Image
Each CIFilter has one or more CIKernel functions

Input to Sepia Hue Contrast Working Space

O
Filter Filter Filter to Output

Original CIImage Output CIImage

kernel vec4 sepia () kernel vec4 hue () kernel vec4 contrast ()

A Very Brief Introduction to Core Image
Multiple CIKernels are concatenated to improve performance

Input to Sepia
Contrast
ConcatenatedHue Working Space
O
FilterProgram
Filter
Filter to Output

Original CIImage Output CIImage

kernel vec4 sepia ()

kernel vec4 hue ()

kernel vec4 contrast ()

Summary of What’s New
Our goal for this release is to enable developers to get better:
Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance
Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance Information
Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance Information Functionality

Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance Information Functionality

Write CIKernels in Metal

CIRenderDestination API
Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance Information Functionality

Write CIKernels in Metal CIRenderInfo API

CIRenderDestination API Xcode Quick Looks
Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance Information Functionality

Write CIKernels in Metal CIRenderInfo API New Filters

CIRenderDestination API Xcode Quick Looks Barcode Support
Depth Support
Summary of What’s New
Our goal for this release is to enable developers to get better:

Performance Information Functionality

Write CIKernels in Metal CIRenderInfo API New Filters

CIRenderDestination API Xcode Quick Looks Barcode Support
Depth Support

Image Editing with Depth WWDC 2017

New Built-In CIFilters
•
196 Built-In Filters

11
196 Built-In Filters
ColorBurnBlendMode DisparityToDepth HueSaturationValueGradient PageCurlTransition
AccordionFoldTransition SoftLightBlendMode
ColorClamp DisplacementDistortion Kaleidoscope PageCurlWithShadowTransition
AdditionCompositing SourceAtopCompositing
ColorControls DissolveTransition LabDeltaE ParallelogramTile
AffineClamp SourceInCompositing
ColorCrossPolynomial DivideBlendMode LanczosScaleTransform PDF417BarcodeGenerator
AffineTile SourceOutCompositing
ColorCube DotScreen LenticularHaloGenerator PerspectiveCorrection
AffineTransform SourceOverCompositing
ColorCubesMixedWithMask Droste LightenBlendMode PerspectiveTile
AreaAverage SpotColor
ColorCubeWithColorSpace EdgePreserveUpsampleFilter LightTunnel PerspectiveTransform
AreaHistogram SpotLight
ColorCurves Edges LinearBurnBlendMode PerspectiveTransformWithExtent
AreaMaximum SRGBToneCurveToLinear
ColorDodgeBlendMode EdgeWork LinearDodgeBlendMode PhotoEffectChrome
AreaMaximumAlpha StarShineGenerator
ColorInvert EightfoldReflectedTile LinearGradient PhotoEffectFade
AreaMinimum StraightenFilter
ColorMap ExclusionBlendMode LinearToSRGBToneCurve PhotoEffectInstant
AreaMinimumAlpha StretchCrop
ColorMatrix ExposureAdjust LineOverlay PhotoEffectMono
AreaMinMaxRed StripesGenerator
ColorMonochrome FalseColor LineScreen PhotoEffectNoir
AttributedTextImageGenerator SubtractBlendMode
ColorPolynomial FlashTransition LuminosityBlendMode PhotoEffectProcess
AztecCodeGenerator SunbeamsGenerator
ColorPosterize FourfoldReflectedTile MaskedVariableBlur PhotoEffectTonal
BarcodeGenerator SwipeTransition
ColumnAverage FourfoldRotatedTile MaskToAlpha PhotoEffectTransfer
BarsSwipeTransition TemperatureAndTint
ComicEffect FourfoldTranslatedTile MaximumComponent PinchDistortion
BicubicScaleTransform TextImageGenerator
ConstantColorGenerator GammaAdjust MaximumCompositing PinLightBlendMode
BlendWithAlphaMask Thermal
Convolution3X3 GaussianBlur MedianFilter Pixellate
BlendWithMask ToneCurve
Convolution5X5 GaussianGradient MinimumComponent Pointillize
Bloom TorusLensDistortion
Convolution7X7 GlassDistortion MinimumCompositing QRCodeGenerator
BokehBlur TriangleKaleidoscope
Convolution9Horizontal GlassLozenge ModTransition RadialGradient
BoxBlur TriangleTile
Convolution9Vertical GlideReflectedTile MorphologyGradient RandomGenerator
BumpDistortion TwelvefoldReflectedTile
CopyMachineTransition Gloom MorphologyMaximum RippleTransition
BumpDistortionLinear TwirlDistortion
Crop HardLightBlendMode MorphologyMinimum RowAverage
CheckerboardGenerator UnsharpMask
Crystallize HatchedScreen MotionBlur SaturationBlendMode
CircleSplashDistortion Vibrance
DarkenBlendMode HeightFieldFromMask MultiplyBlendMode ScreenBlendMode
CircularScreen Vignette
DepthBlurEffect HexagonalPixellate MultiplyCompositing SepiaTone
CircularWrap VignetteEffect
DepthOfField HighlightShadowAdjust NinePartStretched ShadedMaterial
Clamp VortexDistortion
DepthToDisparity HistogramDisplayFilter NinePartTiled SharpenLuminance
CMYKHalftone WhitePointAdjust
DifferenceBlendMode HoleDistortion NoiseReduction SixfoldReflectedTile
Code128BarcodeGenerator XRay
DiscBlur HueAdjust OpTile SixfoldRotatedTile
ColorBlendMode ZoomBlur
DisintegrateWithMaskTransition HueBlendMode OverlayBlendMode SmoothLinearGradient
11
New Built-In CIFilters
Some that are useful for Depth filtering

CIDepthToDisparity / CIDisparityToDepth

CIMorphologyMinimum / Maximum / Gradient

CIColorCubesMixedWithMask

CIAreaMinMaxRed
New Built-In CIFilters
Some that are useful for Depth filtering

CIDepthToDisparity / CIDisparityToDepth

CIMorphologyMinimum / Maximum / Gradient

CIColorCubesMixedWithMask

CIAreaMinMaxRed

CIDepthBlurEffect
New Built-In CIFilters
Some that are useful for Depth filtering

CIDepthToDisparity / CIDisparityToDepth

CIMorphologyMinimum / Maximum / Gradient

CIColorCubesMixedWithMask

CIAreaMinMaxRed

CIDepthBlurEffect

Image Editing with Depth WWDC 2017

New Built-in CIFilters
Some that are often requested additions

CITextImageGenerator

CIColorCurves

CILabDeltaE

CIBicubicScaleTransform

CIBarcodeGenerator
New Built-in CIFilters
Some that are improved

CIHueBlendMode

CISaturationBlendMode

CIColorBlendMode

CILuminosityBlendMode

CILinearBurnBlendMode
New Built-in CIFilters
Some that are improved

CIHueBlendMode

CISaturationBlendMode

CIColorBlendMode

CILuminosityBlendMode

CILinearBurnBlendMode

The demosaic and noise reduction filters used for RAW files
Writing CIKernels in Metal
•

Tony Chu, Core Image Engineer

Writing CIKernels

Input to Sepia
 Hue
 Contrast
 Working Space

O
Filter Filter Filter to Output

Original CIImage Output CIImage

kernel vec4 sepia () kernel vec4 hue () kernel vec4 contrast ()

Writing CIKernels

Input to Sepia
 Hue
 Contrast
 Working Space

O
Filter Filter Filter to Output

Original CIImage Output CIImage

kernel vec4 sepia () kernel vec4 hue () kernel vec4 contrast ()

Writing CIKernels

Previously, kernels are written in CIKernel Language

Writing CIKernels

Previously, kernels are written in CIKernel Language

• Based on GLSL
Writing CIKernels

Previously, kernels are written in CIKernel Language

• Based on GLSL

• Language extensions to enable automatic tiling and subregion rendering

Writing CIKernels

Previously, kernels are written in CIKernel Language

• Based on GLSL

• Language extensions to enable automatic tiling and subregion rendering

destCoord()
Writing CIKernels

Previously, kernels are written in CIKernel Language

• Based on GLSL

• Language extensions to enable automatic tiling and subregion rendering

destCoord()
samplerTransform(sampler src, vec2 p)
Writing CIKernels

Previously, kernels are written in CIKernel Language

• Based on GLSL

• Language extensions to enable automatic tiling and subregion rendering

destCoord()
samplerTransform(sampler src, vec2 p)
sample(sampler src, vec2 p)
Writing CIKernels

Previously, kernels are written in CIKernel Language

• Based on GLSL

• Language extensions to enable automatic tiling and subregion rendering

destCoord()
samplerTransform(sampler src, vec2 p)
sample(sampler src, vec2 p)

• Translated, concatenated, and compiled at run-time to Metal or GLSL

Compiling CIKernels on First Render

Compile CIKernels to Intermediate Representation Render

Translate CIKernels
Concatenate CIKernels
Compile to GPU Code
Compiling CIKernels on First Render

Compile CIKernels to Intermediate Representation Render

Translate CIKernels
Concatenate CIKernels
Compile to GPU Code
Compiling CIKernels on First Render

At build-time (in Xcode)

Compile CIKernels to Intermediate Representation

Render

At run-time

Translate CIKernels
Concatenate CIKernels
Compile to GPU Code
Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Benefits
Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Benefits
• Precompiled at build-time with error diagnostics
Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Benefits
• Precompiled at build-time with error diagnostics

• More modern language features

Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Benefits
• Precompiled at build-time with error diagnostics

• More modern language features

• Still supports concatenation and tiling

Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Benefits
• Precompiled at build-time with error diagnostics

• More modern language features

• Still supports concatenation and tiling

• Can be mixed with traditional CIKernels not written in Metal

Writing CIKernels in Metal NEW

Now, you can write CIKernels directly in Metal Shading Language

Benefits
• Precompiled at build-time with error diagnostics

• More modern language features

• Still supports concatenation and tiling

• Can be mixed with traditional CIKernels not written in Metal

Supported on iOS (for A8 or newer devices), macOS, and tvOS

How to Create Metal CIKernels
 How to Create Metal CIKernels

1. Write CIKernel in Metal shader file

 How to Create Metal CIKernels

1. Write CIKernel in Metal shader file

•

2. Compile and link Metal shader file

 How to Create Metal CIKernels

1. Write CIKernel in Metal shader file

•

2. Compile and link Metal shader file

•

3. Initialize CIKernel with function from Metal library

 How to Create Metal CIKernels

1. Write CIKernel in Metal shader file

•

2. Compile and link Metal shader file

•

3. Initialize CIKernel with function from Metal library

CIKernel Metal Library NEW

New header file containing CIKernel extensions to the Metal Shading Language
CIKernel Metal Library NEW

New header file containing CIKernel extensions to the Metal Shading Language
• CIKernel data types
CIKernel Metal Library NEW

New header file containing CIKernel extensions to the Metal Shading Language
• CIKernel data types
destination, sampler, sample_t
CIKernel Metal Library NEW

New header file containing CIKernel extensions to the Metal Shading Language
• CIKernel data types
destination, sampler, sample_t

• CIKernel functions
CIKernel Metal Library NEW

New header file containing CIKernel extensions to the Metal Shading Language
• CIKernel data types
destination, sampler, sample_t

• CIKernel functions
premultiply, unpremultiply, srgb_to_linear, linear_to_srgb, compare, cossin, sincos
// CIKernelMetalLib.h

namespace coreimage {

struct destination {
float2 coord() const;
};

struct sampler {
float2 transform(float2 p) const;
float2 coord() const;
float4 sample(float2 p) const;
float4 extent() const;
};

// ...
}
// CIKernelMetalLib.h

namespace coreimage {

struct destination {
float2 coord() const;
};

struct sampler {
float2 transform(float2 p) const;
float2 coord() const;
float4 sample(float2 p) const;
float4 extent() const;
};

// ...
}
// CIKernelMetalLib.h

namespace coreimage {

struct destination {
float2 coord() const;
};

struct sampler {
float2 transform(float2 p) const;
float2 coord() const;
float4 sample(float2 p) const;
float4 extent() const;
};

// ...
}
Destination and Samplers

CIKernel Language Metal

Get destination coordinate destCoord() dest.coord()

Transform coordinate to sampler space samplerTransform(src, p) src.transform(p)

Get destination coordinate in sampler space samplerCoord(src) src.coord()

Sample from source image sample(src, p) src.sample(p)

Get extent of source image samplerExtent(src) src.extent()

Destination and Samplers

CIKernel Language Metal

Get destination coordinate destCoord() dest.coord()

Transform coordinate to sampler space samplerTransform(src, p) src.transform(p)

Get destination coordinate in sampler space samplerCoord(src) src.coord()

Sample from source image sample(src, p) src.sample(p)

Get extent of source image samplerExtent(src) src.extent()

// Metal CIWarpKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float2 myWarp (destination dest) {

float2 p = dest.coord();
// do something
return p;
}

}}
// Metal CIWarpKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float2 myWarp (destination dest) {

float2 p = dest.coord();
// do something
return p;
}

}}
// Metal CIWarpKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float2 myWarp (destination dest) {

float2 p = dest.coord();
// do something
return p;
}

}}
// Metal CIWarpKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float2 myWarp (destination dest) {

float2 p = dest.coord();
// do something
return p;
}

}}
// Metal CIWarpKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float2 myWarp (destination dest) { kernel vec2 myWarp () {

float2 p = dest.coord(); vec2 p = destCoord();
// do something // do something
return p; return p;
} }

}}
// Metal CIColorKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float4 myColor (sample_t s) {

// do something
return s;
}

}}
// Metal CIColorKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float4 myColor (sample_t s) {

// do something
return s;
}

}}
// Metal CIColorKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float4 myColor (sample_t s) { kernel vec4 myColor (__sample s) {

// do something // do something
return s; return s;
} }

}}
// Metal CIKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float4 myKernel (sampler src) {

float4 s = src.sample(src.coord());
// do something
return s;
}

}}
// Metal CIKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float4 myKernel (sampler src) {

float4 s = src.sample(src.coord());
// do something
return s;
}

}}
// Metal CIKernel Source

#include <metal_stdlib>
using namespace metal;

#include <CoreImage/CoreImage.h> // includes CIKernelMetalLib.h

extern "C" { namespace coreimage {

float4 myKernel (sampler src) { kernel vec4 myKernel (sampler src) {

float4 s = src.sample(src.coord()); vec4 s = sample(src, samplerCoord(src));
// do something // do something
return s; return s;
} }

}}
 How to Create Metal CIKernels

1. Write CIKernel in Metal shader file

•

2. Compile and link Metal shader file

3. Initialize CIKernel with function from Metal library
•
Compiling and Linking CIKernels

CIKernel Intermediate Metal Library

Source File Representation File Data File

.metal metal .air metallib .metallib

Compiling and Linking CIKernels

CIKernel Intermediate Metal Library

Source File Representation File Data File

.metal metal .air metallib .metallib

Options: -fcikernel
Compiling and Linking CIKernels

CIKernel Intermediate Metal Library

Source File Representation File Data File

.metal metal .air metallib .metallib

Options: -fcikernel -cikernel

xcrun metal -fcikernel MyKernels.metal -o MyKernels.air
xcrun metallib -cikernel MyKernels.air -o MyKernels.metallib
 How to Create Metal CIKernels

1. Write CIKernel in Metal shader file

•

2. Compile and link Metal shader file

•

3. Initialize CIKernel with function from Metal library

New CIKernel API NEW

init(functionName name: String, fromMetalLibraryData data: Data) throws

init(functionName name: String, fromMetalLibraryData data: Data,
outputPixelFormat format: CIFormat) throws
New CIKernel API NEW

init(functionName name: String, fromMetalLibraryData data: Data) throws

init(functionName name: String, fromMetalLibraryData data: Data,
outputPixelFormat format: CIFormat) throws

// Example of initializing CIKernels

let url = Bundle.main.url(forResource: "default", withExtension: "metallib")!

let data = try! Data(contentsOf: url)

let kernel = try? CIKernel(functionName: "myKernel", fromMetalLibraryData: data)

New CIKernel API NEW

init(functionName name: String, fromMetalLibraryData data: Data) throws

init(functionName name: String, fromMetalLibraryData data: Data,
outputPixelFormat format: CIFormat) throws

// Example of initializing CIKernels

let url = Bundle.main.url(forResource: "default", withExtension: "metallib")!

let data = try! Data(contentsOf: url)

let kernel = try? CIKernel(functionName: "myKernel", fromMetalLibraryData: data)

let warpKernel = try? CIWarpKernel(functionName: "myWarp", fromMetalLibraryData: data)
let colorKernel = try? CIColorKernel(functionName: "myColor", fromMetalLibraryData: data)
New CIRenderDestination API
•
New CIRenderDestination API

A consistent API to render to different destination types

• IOSurface

• CVPixelBuffer

• Metal Texture

• OpenGL Texture

• Memory buffer
New CIRenderDestination API

A consistent API to render to different destination types

• IOSurface

• CVPixelBuffer

• Metal Texture

• OpenGL Texture

• Memory buffer

Returns immediately if render fails—with an error

New CIRenderDestination API

Allow setting of common properties for the destination

• Alpha mode behavior

• Clamping mode behavior

• Destination colorspace
New CIRenderDestination API

Allow setting of common properties for the destination

• Alpha mode behavior

• Clamping mode behavior

• Destination colorspace

And some advanced properties

• Dithering

• Blending
New CIRenderDestination API

Allow setting of common properties for the destination

• Alpha mode behavior

• Clamping mode behavior

• Destination colorspace

And some advanced properties

• Dithering

• Blending

Reduces the need for multiple CIContexts

New CIRenderDestination API
Performance Benefits

Existing CIContext APIs for rendering to IOSurface or CVPixelBuffer

• Return to caller when render is completed
New CIRenderDestination API
Performance Benefits

Existing CIContext APIs for rendering to IOSurface or CVPixelBuffer

• Return to caller when render is completed

New CIRenderDestination APIs

• Return to caller when work is issued
New CIRenderDestination API
Performance Benefits

Existing CIContext APIs for rendering to IOSurface or CVPixelBuffer

• Return to caller when render is completed

New CIRenderDestination APIs

• Return to caller when work is issued

This allows CPU and GPU work to be pipelined efficiently

// Clear Then Render Foreground Blended with Background

func renderImagePair (foregroundImage : CIImage, backgroundImage : CIImage,

blend : CIBlendKernel = CIBlendKernel.sourceOver
toSurface : IOSurface)
{
let dest = CIRenderDestination(ioSurface: toSurface)

let ctx = getContext() // don’t create a context each time

try? ctx.startTask(toClear: dest)

try? ctx.startTask(toRender: backgroundImage, to: dest)

dest.blendKernel = blend // use one of 37 built-in blend kernels or create your own

try? ctx.startTask(toRender: foregroundImage, to: dest).waitUntilCompleted()

}
// Clear Then Render Foreground Blended with Background

func renderImagePair (foregroundImage : CIImage, backgroundImage : CIImage,

blend : CIBlendKernel = CIBlendKernel.sourceOver
toSurface : IOSurface)
{
let dest = CIRenderDestination(ioSurface: toSurface)

let ctx = getContext() // don’t create a context each time

try? ctx.startTask(toClear: dest)

try? ctx.startTask(toRender: backgroundImage, to: dest)

dest.blendKernel = blend // use one of 37 built-in blend kernels or create your own

try? ctx.startTask(toRender: foregroundImage, to: dest).waitUntilCompleted()

}
// Clear Then Render Foreground Blended with Background

func renderImagePair (foregroundImage : CIImage, backgroundImage : CIImage,

blend : CIBlendKernel = CIBlendKernel.sourceOver
toSurface : IOSurface)
{
let dest = CIRenderDestination(ioSurface: toSurface)

let ctx = getContext() // don’t create a context each time

try? ctx.startTask(toClear: dest)

try? ctx.startTask(toRender: backgroundImage, to: dest)

dest.blendKernel = blend // use one of 37 built-in blend kernels or create your own

try? ctx.startTask(toRender: foregroundImage, to: dest).waitUntilCompleted()

}
// Clear Then Render Foreground Blended with Background

func renderImagePair (foregroundImage : CIImage, backgroundImage : CIImage,

blend : CIBlendKernel = CIBlendKernel.sourceOver
toSurface : IOSurface)
{
let dest = CIRenderDestination(ioSurface: toSurface)

let ctx = getContext() // don’t create a context each time

try? ctx.startTask(toClear: dest)

try? ctx.startTask(toRender: backgroundImage, to: dest)

dest.blendKernel = blend // use one of 37 built-in blend kernels or create your own

try? ctx.startTask(toRender: foregroundImage, to: dest).waitUntilCompleted()

}
// Clear Then Render Foreground Blended with Background

func renderImagePair (foregroundImage : CIImage, backgroundImage : CIImage,

blend : CIBlendKernel = CIBlendKernel.sourceOver
toSurface : IOSurface)
{
let dest = CIRenderDestination(ioSurface: toSurface)

let ctx = getContext() // don’t create a context each time

try? ctx.startTask(toClear: dest)

try? ctx.startTask(toRender: backgroundImage, to: dest)

dest.blendKernel = blend // use one of 37 built-in blend kernels or create your own

try? ctx.startTask(toRender: foregroundImage, to: dest).waitUntilCompleted()

}
// Clear Then Render Foreground Blended with Background

func renderImagePair (foregroundImage : CIImage, backgroundImage : CIImage,

blend : CIBlendKernel = CIBlendKernel.sourceOver
toSurface : IOSurface)
{
let dest = CIRenderDestination(ioSurface: toSurface)

let ctx = getContext() // don’t create a context each time

try? ctx.startTask(toClear: dest)

try? ctx.startTask(toRender: backgroundImage, to: dest)

dest.blendKernel = blend // use one of 37 built-in blend kernels or create your own

try? ctx.startTask(toRender: foregroundImage, to: dest).waitUntilCompleted()

}
Rendering to Metal Drawable Textures

let ctx = getContext() // don’t create a context each time

let texture = currentDrawable.texture

let dest = CIRenderDestination(texture, commandBuffer: cmdBuffer)

try? ctx.startTask(toRender: image, to: dest)

Rendering to Metal Drawable Textures

let ctx = getContext() // don’t create a context each time

let texture = currentDrawable.texture

let dest = CIRenderDestination(texture, commandBuffer: cmdBuffer)

try? ctx.startTask(toRender: image, to: dest)

Rendering to Metal Drawable Textures

let ctx = getContext() // don’t create a context each time

let texture = currentDrawable.texture

let dest = CIRenderDestination(texture, commandBuffer: cmdBuffer)

try? ctx.startTask(toRender: image, to: dest)

Rendering to Metal Drawable Textures

let ctx = getContext() // don’t create a context each time

while stillDrawing {
let texture = currentDrawable.texture

let dest = CIRenderDestination(texture, commandBuffer: cmdBuffer)

try? ctx.startTask(toRender: image, to: dest)

}
 Rendering to Metal Drawable Textures

CPU

GPU
 Rendering to Metal Drawable Textures

CPU

GPU

Get drawable
 Rendering to Metal Drawable Textures

CPU

GPU

Get drawable
Call startTask
 Rendering to Metal Drawable Textures

Frame A
CPU I1 I2 D

Frame A
GPU
I1 I2 D

Get drawable
Call startTask
 Rendering to Metal Drawable Textures

Frame A
CPU I1 I2 D

Frame A
GPU
I1 I2 D

Get drawable
Call startTask

startTask returns
 Rendering to Metal Drawable Textures

Frame A
CPU I1 I2 D

Frame A
GPU
I1 I2 D

Get drawable Get drawable

Call startTask

startTask returns
 Rendering to Metal Drawable Textures

Frame A
CPU I1 I2 D

Frame A
GPU
I1 I2 D

Get drawable Get drawable

Call startTask

startTask returns
 Rendering to Metal Drawable Textures

Frame A Frame B
CPU I1 I2 D I1 I2 D

Frame A Frame B
GPU
I1 I2 D I1 I2 D

Get drawable Get drawable

Call startTask Call startTask

startTask returns
 Rendering to Metal Drawable Textures

Frame A Frame B Frame C

CPU I1 I2 D I1 I2 D I1 I2 D

Frame A Frame B Frame C

GPU
I1 I2 D I1 I2 D I1 I2 D

Get drawable Get drawable

Call startTask Call startTask

startTask returns
Rendering to Metal Drawable Textures Improved

let ctx = getContext() // don’t create a context each time

while stillDrawing {
let dest = CIRenderDestination(
width: 1024,
height: 768,
pixelFormat: MTLPixelFormat.rgba8Unorm,
commandBuffer: cmdBuffer) { () -> MTLTexture in
return currentDrawable.texture
})

try? ctx.startTask(toRender: image, to: dest)

}
Rendering to Metal Drawable Textures Improved

let ctx = getContext() // don’t create a context each time

while stillDrawing {
let dest = CIRenderDestination(
width: 1024,
height: 768,
pixelFormat: MTLPixelFormat.rgba8Unorm,
commandBuffer: cmdBuffer) { () -> MTLTexture in
return currentDrawable.texture
})

try? ctx.startTask(toRender: image, to: dest)

}
Rendering to Metal Drawable Textures Improved

let ctx = getContext() // don’t create a context each time

while stillDrawing {
let dest = CIRenderDestination(
width: 1024,
height: 768,
pixelFormat: MTLPixelFormat.rgba8Unorm,
commandBuffer: cmdBuffer) { () -> MTLTexture in
return currentDrawable.texture
})

try? ctx.startTask(toRender: image, to: dest)

}
Rendering to Metal Drawable Textures Improved

let ctx = getContext() // don’t create a context each time

while stillDrawing {
let dest = CIRenderDestination(
width: 1024,
height: 768,
pixelFormat: MTLPixelFormat.rgba8Unorm,
commandBuffer: cmdBuffer) { () -> MTLTexture in
return currentDrawable.texture
})

try? ctx.startTask(toRender: image, to: dest)

}
 Rendering to Metal Drawable Textures Improved

Frame A Frame B Frame C

CPU I1 I2 D I1 I2 D I1 I2 D

Frame A Frame B Frame C

GPU
I1 I2 D I1 I2 D I1 I2 D
 Rendering to Metal Drawable Textures Improved

Frame A Frame B Frame C

CPU I1 I2 D I1 I2 D I1 I2 D

Frame A Frame B Frame C

GPU
I1 I2 D I1 I2 D I1 I2 D
Looking Inside Core Image
•

Quick Look support and other runtime information

•
Detailed Image and Render Information
See how Core Image works internally

CIRenderTask returns a CIRenderInfo object when complete

passCount
kernelExecutionTime
kernelPixelsProcessed

45
Detailed Image and Render Information
Awesome Core Image Quick Looks in Xcode

46
Detailed Image and Render Information
Awesome Core Image Quick Looks in Xcode

CIImage Shows the image graph as constructed

46
Detailed Image and Render Information
Awesome Core Image Quick Looks in Xcode

CIImage Shows the image graph as constructed

CIRenderTask Shows how Core Image optimized the graph

46
Detailed Image and Render Information
Awesome Core Image Quick Looks in Xcode

CIImage Shows the image graph as constructed

CIRenderTask Shows how Core Image optimized the graph

CIRenderInfo Shows concatenation, timing, and caching

46
CIImage Quick Look

// load image and apply orientation

var image = CIImage(
contentsOf: url
options: [kCIImageApplyOrientationProperty: true])

// downsample by 2x
image = image.applying(CGAffineTransform(scaleX:0.5, y:0.5))
CIImage Quick Look

// load image and apply orientation

var image = CIImage(
contentsOf: url
options: [kCIImageApplyOrientationProperty: true])

// downsample by 2x
image = image.applying(CGAffineTransform(scaleX:0.5, y:0.5))
(CIImage)0x0000000100e61630
CIImage Quick Look

// load image and apply orientation

var image = CIImage(
contentsOf: url
options: [kCIImageApplyOrientationProperty: true])

// downsample by 2x
image = image.applying(CGAffineTransform(scaleX:0.5, y:0.5))
(CIImage)0x0000000100e61630

CIImage <0x100e61630>

affine
0.5 0 0
0 0.5 0
extent=[0 0 1512 2016]
opaque

affine
0 -1 0
1 0 4032
extent=[0 0 3024 4032]
opaque

affine
1 0 0
0 -1 3024
extent=[0 0 4032 3024]
opaque

colormatch
Display P3_to_workingspace
extent=[0 0 4032 3024]
opaque

IOSurface 0x100f483d0(189) seed:1 YCC420f 601 alpha_one

extent=[0 0 4032 3024]
opaque
 CIImage <0x100e61630>

affine
0.5 0 0
0 0.5 0
extent=[0 0 1512 2016]
opaque

affine
0 -1 0
1 0 4032
extent=[0 0 3024 4032]
opaque

affine
1 0 0
0 -1 3024
extent=[0 0 4032 3024]
opaque

colormatch
Display P3_to_workingspace
extent=[0 0 4032 3024]
opaque

IOSurface 0x100f483d0(189) seed:1 YCC420f 601 alpha_one

extent=[0 0 4032 3024]
opaque
 CIImage <0x100e61630>
affine
0 -1 0
1 0 4032
extent=[0
affine 0 3024 4032]
opaque
0.5 0 0
0 0.5 0
extent=[0 0 1512 2016]
opaque

affine
affine
0 -1 0
1 01 00 4032
0 -1 30240 3024 4032]
extent=[0
opaque
extent=[0 0 4032 3024]
opaque

affine
1 0 0
0 -1 3024
extent=[0 0 4032 3024]
opaque
colormatch
Display P3_to_workingspace
extent=[0 0 4032 3024]
opaque
colormatch
Display P3_to_workingspace
extent=[0 0 4032 3024]
opaque

IOSurface IOSurface
0x100f483d0(418) seed:1 YCC420f 601 alpha_one
0x100f483d0(189) seed:1 YCC420f 601 alpha_one
extent=[0 extent=[0
0 40320 3024]
4032 3024]
opaque
opaque
 opaque

CIImage <0x100e61630>

affine
affine
0 -10.5 00 0
1 00 0.540320
extent=[0 0 1512 2016]
extent=[0
opaque 0 3024 4032]
opaque

affine
0 -1 0
1 0 4032
extent=[0 0 3024 4032]
affine
opaque

1 0 0
0 -1 3024
extent=[0 0 4032 3024]
affine
opaque
1 0 0
0 -1 3024
extent=[0 0 4032 3024]
opaque

colormatch
colormatch
Display P3_to_workingspace
Display P3_to_workingspace
extent=[0 0 4032 3024]
extent=[0 0 4032 3024]
opaque
opaque

IOSurface 0x100f483d0(189) seed:1 YCC420f 601 alpha_one

extent=[0 0 4032 3024]
opaque

IOSurface 0x1017ac560(276) seed:1 YCC420f 601 alpha_one

extent=[0 0 4032 3024]
opaque
 affine
0.5 CIImage
0 0 <0x100e61630>
0 0.5 0
affine
extent=[0
0.5 0 0 0 1512 2016]
affine
opaque
0 0.5 0
0.5 0 0 0 1512 2016]
extent=[0
affine
0opaque
0.5
0.5 0 00
0 0.5 0
extent=[0 0 1512 2016]
extent=[0 0 1512 2016]
opaque
opaque

affine
0 -1 0
1 0 4032
affine
affine
0 -1
0 -1 0 0 0 3024 4032]
extent=[0
1 01 04032
opaque 4032
affine
extent=[0 0 3024 4032]
extent=[0 0 3024 4032]
0 -1 0
opaque
opaque
1 0 4032
extent=[0 0 3024 4032]
opaque
affine
affine
1 0 0
1 00 -10 3024
affine
0 -1 30240 4032 3024]
extent=[0
opaque
1 0 0
extent=[0 0 4032 3024]
0 -1 3024
opaque
extent=[0 0 4032 3024]
affine
1opaque
0 0
colormatch
0 Display
-1 3024P3_to_workingspace
extent=[0 0 4032 3024]
extent=[0
opaque
0 4032 3024]
opaque
colormatch
Display P3_to_workingspace
colormatch0 4032 3024]
extent=[0
Display
IOSurface opaque P3_to_workingspace
0x100f483d0(189) seed:1 YCC420f 601 alpha_one
extent=[0 extent=[0
0 4032 3024] 0 4032 3024]
opaque
opaque
colormatch
Display P3_to_workingspace
extent=[0 0 4032 3024]
IOSurface 0x1017ac560(276) seed:1 YCC420f 601 alpha_one
CIImage Quick Look
// load disparity image and apply orientation
var disparity = CIImage(
contentsOf: url,
options: [kCIImageAuxiliaryDisparity: true,
kCIImageApplyOrientationProperty: true])

// cubic upsample by 2x
disparity = disparity.applyingFilter(“CIBicubicScaleTransform”, withInputParameters: […])

// adjust disparity mask and blend foreground and background

let mask = disparity.applyingFilter(“CIColorControls",
withInputParameters: [kCIInputContrastKey: 2.0])
mask = mask.applyingFilter(“CIColorClamp")
CIImage Quick Look
// load disparity image and apply orientation
var disparity = CIImage(
contentsOf: url,
options: [kCIImageAuxiliaryDisparity: true,
kCIImageApplyOrientationProperty: true])

// cubic upsample by 2x
disparity = disparity.applyingFilter(“CIBicubicScaleTransform”, withInputParameters: […])
(CIImage) 0x0000000100f29480
// adjust disparity mask and blend foreground and background
let mask = disparity.applyingFilter(“CIColorControls",
withInputParameters: [kCIInputContrastKey: 2.0])
mask = mask.applyingFilter(“CIColorClamp")
CIImage Quick Look
// load disparity image and apply orientation
var disparity = CIImage(
contentsOf: url,
options: [kCIImageAuxiliaryDisparity: true,
kCIImageApplyOrientationProperty: true])

// cubic upsample by 2x
disparity = disparity.applyingFilter(“CIBicubicScaleTransform”, withInputParameters: […])
(CIImage) 0x0000000100f29480
// adjust disparity mask and blend foreground and background CIImage <0x100f29480>

let mask = disparity.applyingFilter(“CIColorControls", kernel _cubicUpsample

src
scale=[0.380952 0.380952 0 0]
coefsLT1=[1.5 -2.5 0 1]
coefsLT2=[-0.5 2.5 -4 2]
extent=[-6 -6 1524 2028]

withInputParameters: [kCIInputContrastKey: 2.0]) affine

0 -1 0
1 0 768
extent=[0 0 576 768]

mask = mask.applyingFilter(“CIColorClamp")
opaque

affine
1 0 0
0 -1 576
extent=[0 0 768 576]
opaque

colormatch
Linear Gray_to_workingspace
extent=[0 0 768 576]
opaque

IOSurface 0x100e720c0(218) seed:1 Lh alpha_one

extent=[0 0 768 576]
opaque
 CIImage <0x100f29480>

kernel _cubicUpsample
src
scale=[0.380952 0.380952 0 0]
coefsLT1=[1.5 -2.5 0 1]
coefsLT2=[-0.5 2.5 -4 2]
extent=[-6 -6 1524 2028]

affine
0 -1 0
1 0 768
extent=[0 0 576 768]
opaque

affine
1 0 0
0 -1 576
extent=[0 0 768 576]
opaque

colormatch
Linear Gray_to_workingspace
extent=[0 0 768 576]
opaque

IOSurface 0x100e720c0(218) seed:1 Lh alpha_one

extent=[0 0 768 576]
opaque
 CIImage <0x100f29480>

affine
0 -1 0
1 0 768
kernel _cubicUpsample
extent=[0
src 0 576 768]
opaque
scale=[0.380952 0.380952 0 0]
coefsLT1=[1.5 -2.5 0 1]
coefsLT2=[-0.5 2.5 -4 2]
extent=[-6 -6 1524 2028]

affine
affine
0 -1 0
1 01 00 768
0 -1 5760 576 768]
extent=[0
opaque
extent=[0 0 768 576]
opaque

affine
1 0 0
0 -1 576
extent=[0 0 768 576]
opaque
colormatch
Linear Gray_to_workingspace
extent=[0 0 768 576]
opaque
colormatch
Linear Gray_to_workingspace
extent=[0 0 768 576]
opaque

IOSurface 0x100e720c0(428) seed:1 Lh alpha_one

IOSurface 0x100e720c0(218) seed:1 Lh alpha_one
extent=[0 0 7680 768
extent=[0 576]
576]
opaque
opaque
 CIImage <0x100f29480>

kernel _cubicUpsample
src kernel _cubicUpsample
src
scale=[0.380952 0.380952
scale=[0.380952 0.380952 0 0] 0 0]
coefsLT1=[1.5 -2.5 0 1]
coefsLT1=[1.5 -2.5 0 1]
coefsLT2=[-0.5 2.5 -4 2]
coefsLT2=[-0.5 2.52028]
extent=[-6 -6 1524 -4 2]
extent=[-6 -6 1524 2028]

affine
0 -1 0
1 0 768
extent=[0 0 576 768]
affine
opaque

0 -1 0
1 0 768
extent=[0 0 576 768]
affine
opaque
1 0 0
0 -1 576
extent=[0 0 768 576]
opaque

affine
colormatch
1Linear
0 Gray_to_workingspace
0
0extent=[0
-1 576 0 768 576]
opaque
extent=[0 0 768 576]
opaque

IOSurface 0x100e720c0(218) seed:1 Lh alpha_one

extent=[0 0 768 576]
opaque

colormatch
Linear Gray_to_workingspace
extent=[0 0 768 576]
 CIImage <0x100f29480>

kernel _cubicUpsample
src
scale=[0.380952 0.380952 0 0]
coefsLT1=[1.5 -2.5 0 1]
coefsLT2=[-0.5 2.5 -4 2]
extent=[-6 -6 1524 2028]

affine
0 -1 0
1 0 768
extent=[0 0 576 768]
opaque

affine
1 0 0
0 -1 576
extent=[0 0 768 576]
opaque

colormatch
Linear Gray_to_workingspace
extent=[0 0 768 576]
opaque

IOSurface 0x100e720c0(218) seed:1 Lh alpha_one

extent=[0 0 768 576]
opaque
CIImage Quick Look
// adjustment for foreground
let fg = image.applyingFilter("CIPhotoEffectFade")

// adjustment for background

let bg = image.applyingFilter("CIPhotoEffectNoir")

// combine foreground and background using mask

let output = fg.applyingFilter("CIBlendWithMask",
withInputParameters: [ kCIInputBackgroundImageKey: bg,
kCIInputMaskImageKey: mask ])
CIImage Quick Look
// adjustment for foreground
let fg = image.applyingFilter("CIPhotoEffectFade")

// adjustment for background

let bg = image.applyingFilter("CIPhotoEffectNoir")

// combine foreground and background using mask

let output = fg.applyingFilter("CIBlendWithMask",
(CIImage) 0x0000000100e341d0
withInputParameters: [ kCIInputBackgroundImageKey: bg,
kCIInputMaskImageKey: mask ])
CIImage Quick Look
// adjustment for foreground
let fg = image.applyingFilter("CIPhotoEffectFade")

// adjustment for background

let bg = image.applyingFilter("CIPhotoEffectNoir")

// combine foreground and background using mask CIImage <0x100e341d0>

colorkernel _blendWithMask
f=image_0

let output = fg.applyingFilter("CIBlendWithMask",

b=image_1
m=image_2
extent=[0 0 1512 2016]

0 1

color_matrix color_matrix
r=(1.51572 0 0 0) r=(0.659754 0 0 0)

(CIImage) 0x0000000100e341d0
2
g=(0 1.51572 0 0) g=(0 0.659754 0 0)

withInputParameters: [ kCIInputBackgroundImageKey: bg,

b=(0 0 1.51572 0) b=(0 0 0.659754 0) premultiply
a=(0 0 0 1) a=(0 0 0 1) extent=[-6 -6 1524 2028]

extent=[0 0 1512 2016] extent=[0 0 1512 2016]

opaque opaque

colorkernel _colorClamp
colormatch colormatch
c

kCIInputMaskImageKey: mask ])
devicergb_to_workingspace devicergb_to_workingspace
lo=[0 0 0 0]
extent=[0 0 1512 2016] extent=[0 0 1512 2016]
hi=[1 1 1 1]
opaque opaque
extent=[-6 -6 1524 2028]

colorkernel _colorcubeopaque_denorm colorkernel _colorcubeopaque_denorm

im=image_3 im=image_5
cube=image_4 cube=image_6 unpremultiply
dims=[31 0.0322581 0.03125 0.000976562] dims=[31 0.0322581 0.03125 0.000976562] extent=[-6 -6 1524 2028]
extent=[0 0 1512 2016] extent=[0 0 1512 2016]
opaque opaque

3 4
colormatch
workingspace_to_devicergb
extent=[0 0 1512 2016]
opaque
color_matrix
5 6 r=(2 0 0 0)
g=(0 2 0 0)
provider BGRA8 32x1024 colormatch provider BGRA8 32x1024
b=(0 0 2 0)
alpha_one edge_clamp cache workingspace_to_devicergb alpha_one edge_clamp cache
a=(0 0 0 1)
extent=[infinite][0 0 32 1024] extent=[0 0 1512 2016] extent=[infinite][0 0 32 1024]
bias=(-0.5 -0.5 -0.5 0)
opaque opaque opaque

extent=[-6 -6 1524 2028]

kernel _cubicUpsample
affine
src
0.5 0 0
scale=[0.380952 0.380952 0 0]
0 0.5 0
coefsLT1=[1.5 -2.5 0 1]
extent=[0 0 1512 2016]
coefsLT2=[-0.5 2.5 -4 2]
opaque
extent=[-6 -6 1524 2028]

affine affine
0 -1 0 0 -1 0
1 0 4032 1 0 768
extent=[0 0 3024 4032] extent=[0 0 576 768]
opaque opaque

affine affine
1 0 0 1 0 0
0 -1 3024 0 -1 576
extent=[0 0 4032 3024] extent=[0 0 768 576]
opaque opaque

colormatch colormatch
Display P3_to_workingspace Linear Gray_to_workingspace
extent=[0 0 4032 3024] extent=[0 0 768 576]
opaque opaque

IOSurface 0x100e0aa10(29) seed:1 YCC420f 601 alpha_one IOSurface 0x100e17c00(213) seed:1 Lh alpha_one
extent=[0 0 4032 3024] extent=[0 0 768 576]
opaque opaque
 CIImage <0x10353a570>

CIImage Quick Look colorkernel _blendWithMask

f=image_0
b=image_1
m=image_2
extent=[0 0 1512 2016]

// adjustment for foreground

0 1
let fg = image.applyingFilter("CIPhotoEffectFade") 2
colormatch colormatch

fg = fg.applyingFilter(“CIExposureAdjust", withInputParameters: [kCIInputEVKey : 0.6])

devicergb_to_workingspace devicergb_to_workingspace premultiply
extent=[0 0 1512 2016] extent=[0 0 1512 2016] extent=[-6 -6 1524 2028]
opaque opaque

// adjustment for background

colorkernel _colorcubeopaque_denorm colorkernel _colorcubeopaque_denorm
let bg = image.applyingFilter("CIPhotoEffectNoir")
im=image_3 im=image_5
colorkernel _colorClamp
c
cube=image_4 cube=image_6
lo=[0 0 0 0]
bg = bg.applyingFilter("CIExposureAdjust", withInputParameters: [kCIInputEVKey : -0.6])
dims=[31 0.0322581 0.03125 0.000976562]
extent=[0 0 1512 2016]
dims=[31 0.0322581 0.03125 0.000976562]
extent=[0 0 1512 2016]
hi=[1 1 1 1]
extent=[-6 -6 1524 2028]
opaque opaque

// combine foreground and background using mask

3 4 5 6

let output
colormatch = fg.applyingFilter("CIBlendWithMask",
provider BGRA8 32x1024 colormatch provider BGRA8 32x1024
unpremultiply
workingspace_to_devicergb alpha_one edge_clamp cache workingspace_to_devicergb alpha_one edge_clamp cache
extent=[-6 -6 1524 2028]
withInputParameters: [ kCIInputBackgroundImageKey: bg,
extent=[0 0 1512 2016]
opaque
extent=[infinite][0 0 32 1024]
opaque
extent=[0 0 1512 2016]
opaque
extent=[infinite][0 0 32 1024]
opaque

kCIInputMaskImageKey: mask ])

color_matrix
r=(2 0 0 0)
affine
g=(0 2 0 0)
0.5 0 0
b=(0 0 2 0)
0 0.5 0
a=(0 0 0 1)
extent=[0 0 1512 2016]
bias=(-0.5 -0.5 -0.5 0)
opaque

extent=[-6 -6 1524 2028]

 CIImage <0x10353a570>
<0x1017c9630>

CIImage Quick Look _blendWithMask

colorkernel _blendWithMask
f=image_0
b=image_1
m=image_2
extent=[0 0 1512 2016]
2016]

// adjustment for foreground

0 1
let fg = image.applyingFilter("CIPhotoEffectFade")
colormatch colormatch
22
colormatch colormatch

fg = fg.applyingFilter(“CIExposureAdjust", withInputParameters: [kCIInputEVKey : 0.6])

devicergb_to_workingspace devicergb_to_workingspace
devicergb_to_workingspace premultiply
premultiply
extent=[0 0 1512 2016] 2016]
extent=[0 0 1512 2016] extent=[-6 -6
extent=[-6 -6 1524
1524 2028]
2028]
opaque opaque

// adjustment for background

colorkernel _colorcubeopaque_denorm _colorcubeopaque_denorm
colorkernel _colorcubeopaque_denorm
let bg = image.applyingFilter("CIPhotoEffectNoir")
im=image_3 im=image_5
colorkernel _colorClamp
colorkernel
cc
_colorClamp

cube=image_4 cube=image_6
lo=[0 00 00 0]
lo=[0 0]
bg = bg.applyingFilter("CIExposureAdjust", withInputParameters: [kCIInputEVKey : -0.6])
dims=[31 0.0322581 0.03125 0.000976562]
extent=[0 0 1512 2016] extent=[0 0 1512 2016]
0.000976562]
dims=[31 0.0322581 0.03125 0.000976562]
hi=[1 11 11 1]
hi=[1 1]
extent=[-6 -6
extent=[-6 -6 1524
1524 2028]
2028]
opaque opaque

// combine foreground and background using mask

3 4 5 66

let output
colormatch = fg.applyingFilter("CIBlendWithMask",
provider BGRA8 32x1024 colormatch provider BGRA8
provider BGRA8 32x1024
32x1024
unpremultiply
unpremultiply
workingspace_to_devicergb alpha_one edge_clamp cache workingspace_to_devicergb alpha_one edge_clamp
alpha_one edge_clamp cache
cache
extent=[-6 -6
extent=[-6 -6 1524
1524 2028]
2028]
withInputParameters: [ kCIInputBackgroundImageKey: bg,
extent=[0 0 1512 2016]
opaque
extent=[infinite][0 0 32 1024]
opaque
extent=[0 0 1512 2016]
opaque
extent=[infinite][0 00 32
extent=[infinite][0
opaque
opaque
32 1024]
1024]

kCIInputMaskImageKey: mask ])

color_matrix
color_matrix
r=(2 00 00 0)
r=(2 0)
affine
g=(0 22 00 0)
g=(0 0)
0.5 0 0
b=(0 00 22 0)
b=(0 0)
0 0.5 0
a=(0 00 00 1)
a=(0 1)
extent=[0 0 1512 2016]
bias=(-0.5 -0.5
bias=(-0.5 -0.5 -0.5
-0.5 0)
0)
opaque

extent=[-6 -6
extent=[-6 -6 1524
1524 2028]
2028]
 CIImage
CIImage <0x1017c9630>
<0x10353a570>

CIImage Quick Look colorkernel _blendWithMask

f=image_0
b=image_1
m=image_2
extent=[0 0 1512 2016]

// adjustment for foreground

00 1
let fg = image.applyingFilter("CIPhotoEffectFade")
colormatch colormatch
2
colormatch colormatch
devicergb_to_workingspace devicergb_to_workingspace
fg = fg.applyingFilter(“CIExposureAdjust", withInputParameters: [kCIInputEVKey : 0.6])
devicergb_to_workingspace premultiply
extent=[0
extent=[0 00 1512
1512 2016]
2016] extent=[0 0 1512 2016] 2028]
extent=[-6 -6 1524 2028]
opaque
opaque opaque

// adjustment for background

colorkernel
colorkernel _colorcubeopaque_denorm
_colorcubeopaque_denorm colorkernel _colorcubeopaque_denorm
let bg = image.applyingFilter("CIPhotoEffectNoir")
im=image_3
im=image_3 im=image_5
c
_colorClamp
colorkernel _colorClamp

cube=image_4
cube=image_4 cube=image_6
lo=[0 0 0 0]
bg = bg.applyingFilter("CIExposureAdjust", withInputParameters: [kCIInputEVKey : -0.6])
dims=[31
dims=[31 0.0322581
extent=[0
0.0322581 0.03125
extent=[0 00 1512
0.03125 0.000976562]
1512 2016]
2016]
0.000976562] dims=[31 0.0322581 0.03125 0.000976562]
extent=[0 0 1512 2016]
hi=[1 1 1 1]
2028]
extent=[-6 -6 1524 2028]
opaque
opaque opaque

// combine foreground and background using mask

33 44 5 6

let colormatch
output
colormatch = fg.applyingFilter("CIBlendWithMask",
provider
provider BGRA8
BGRA8 32x1024
32x1024 colormatch provider BGRA8 32x1024
unpremultiply
workingspace_to_devicergb
workingspace_to_devicergb alpha_one
alpha_one edge_clamp
edge_clamp cache
cache workingspace_to_devicergb alpha_one edge_clamp cache
2028]
extent=[-6 -6 1524 2028]
withInputParameters: [ kCIInputBackgroundImageKey: bg,
extent=[0
extent=[0 00 1512
opaque
opaque
1512 2016]
2016] extent=[infinite][0
extent=[infinite][0 00 32
opaque
opaque
32 1024]
1024] extent=[0 0 1512 2016]
opaque
extent=[infinite][0 0 32 1024]
opaque

kCIInputMaskImageKey: mask ])

color_matrix
r=(2 0 0 0)
affine
affine
g=(0 2 0 0)
0.5 00 00
0.5
b=(0 0 2 0)
00 0.5
0.5 00
a=(0 0 0 1)
extent=[0 00 1512
extent=[0 1512 2016]
2016]
bias=(-0.5 -0.5 -0.5 0)
0)
opaque
opaque

extent=[-6 -6 1524 2028]

2028]
 CIImage <0x10353a570>

CIImage Quick Look colorkernel

colorkernel _blendWithMask
f=image_0
f=image_0
b=image_1
_blendWithMask

b=image_1
m=image_2
extent=[0 0 1512 2016]
m=image_2
extent=[0 0 1512 2016]
// adjustment for foreground
0 1
let fg = image.applyingFilter("CIPhotoEffectFade") 2
colormatch colormatch

fg = fg.applyingFilter(“CIExposureAdjust", withInputParameters: [kCIInputEVKey : 0.6])

devicergb_to_workingspace devicergb_to_workingspace premultiply
extent=[0 0 1512 2016] extent=[0 0 1512 2016] extent=[-6 -6 1524 2028]
0 opaque opaque 1

color_matrix color_matrix
//0 adjustment
r=(1.51572 0 0) for background r=(0.659754 0 0 0)
2
g=(0 1.51572 0 0) colorkernel _colorcubeopaque_denorm g=(0 0.659754
colorkernel 0 0)
_colorcubeopaque_denorm
let bg
b=(0 0 1.51572 0) = im=image_3
image.applyingFilter("CIPhotoEffectNoir") b=(0 0 0.659754 0)
im=image_5
colorkernel _colorClamp
c premultiply
cube=image_4 cube=image_6
a=(0 0 0 1) a=(00.0322581
0 0 1)0.03125 0.000976562] lo=[0 0 0 0] extent=[-6 -6 1524 2028]
bg = bg.applyingFilter("CIExposureAdjust",
extent=[0 0 1512 2016]
withInputParameters: [kCIInputEVKey : -0.6])
dims=[31 0.0322581 0.03125 0.000976562] dims=[31
extent=[0 0 1512 2016]
hi=[1 1 1 1]
extent=[-6 -6 1524 2028]
opaque opaque
extent=[0 0 1512 2016] extent=[0 0 1512 2016]
opaque opaque
// combine foreground and background using mask
3 4 5 6

let output
colormatch = fg.applyingFilter("CIBlendWithMask",
provider BGRA8 32x1024 colormatch provider BGRA8 32x1024
unpremultiply
workingspace_to_devicergb alpha_one edge_clamp cache workingspace_to_devicergb alpha_one edge_clamp cache
extent=[-6 -6 1524 2028]
withInputParameters: [ kCIInputBackgroundImageKey: bg,
extent=[0 0 1512 2016]
opaque
extent=[infinite][0 0 32 1024]
opaque
extent=[0 0 1512 2016]
opaque
extent=[infinite][0 0 32 1024]
opaque
colorkernel _colorClamp
olormatch kCIInputMaskImageKey:
colormatch mask ]) c
evicergb_to_workingspace devicergb_to_workingspace
lo=[0 0 0 0]
xtent=[0 0 1512 2016] extent=[0 0 1512 2016]
color_matrix hi=[1 1 1 1]
paque opaque r=(2 0 0 0)
affine
g=(0 2 0 0)
extent=[-6 -6 1524 2028]
0.5 0 0
b=(0 0 2 0)
0 0.5 0
a=(0 0 0 1)
extent=[0 0 1512 2016]
bias=(-0.5 -0.5 -0.5 0)
opaque

extent=[-6 -6 1524 2028]

CIRenderTask Quick Look
// render output to an IOSurface
let dest = CIRenderDestination(ioSurface: surface);

let context = self.context;

guard
let task = try? context.startTask(toRender: output, to: dest),
let info = try? task.waitUntilCompleted()
else {
// handle render failure
}
CIRenderTask Quick Look
// render output to an IOSurface
let dest = CIRenderDestination(ioSurface: surface);

let context = self.context;

guard
let task = try? context.startTask(toRender: output, to: dest),
let info = try? task.waitUntilCompleted()
(CIRenderTask) 0x0000000102806b20
else {
// handle render failure
}
 optimized graph
render_to_surface
(metal context 1 frame 1)
format=BGRA8 roi=[0 0 1512 2016]

CIRenderTask Quick Look

affine
1 0 0
0 -1 2016
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=114BF1F26BC2E310

clamp_to_alpha
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=7FF7DA85C4C272F

premultiply
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=C1C4C4605B7A67F8

linear_to_srgb
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]

// render output to an IOSurface

digest=28A3941A3DEF8ED3

unpremultiply
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=6E8216B0766EE943

let dest = CIRenderDestination(ioSurface: surface); colorkernel _blendWithMask

f=(0)
b=(1)
m=(2)
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=A03F1302AE52FA2

0 1 2

premultiply premultiply premultiply

rois=[0 0 1512 2016] rois=[0 0 1512 2016] rois=[0 0 1512 2016]
extent=[infinite][empty] extent=[infinite][empty] extent=[-6 -6 1524 2028]

let context = self.context;

digest=35A1ED32B7CF9623 digest=28E64EF30464138A digest=A151E67FC910233

colorkernel _colorClamp
c
srgb_to_linear srgb_to_linear
lo=[0 0 0 0]
rois=[0 0 1512 2016] rois=[0 0 1512 2016]
hi=[1 1 1 1]
extent=[infinite][empty] extent=[infinite][empty]
rois=[0 0 1512 2016]
digest=EAD1A08A4B0E2AFF digest=EE71EA0E37C9BACA
extent=[-6 -6 1524 2028]
digest=74289F843E3F4F71

unpremultiply unpremultiply unpremultiply

rois=[0 0 1512 2016] rois=[0 0 1512 2016] rois=[0 0 1512 2016]

guard
extent=[infinite][empty] extent=[infinite][empty] extent=[-6 -6 1524 2028]
digest=5888021ED6FE2240 digest=59D03685D7B281C8 digest=FCCD5FFF99FAFE12

color_matrix_3x4
colorkernel _colorcubeopaque_denorm colorkernel _colorcubeopaque_denorm
r=[2 0 0]
im=(3) im=(5)
g=[0 2 0]
cube=(4) cube=(6)

let task = try? context.startTask(toRender: output, to: dest),

b=[0 0 2]
dims=[31 0.0322581 0.03125 0.000976562] dims=[31 0.0322581 0.03125 0.000976562]
bias=[-0.5 -0.5 -0.5]
rois=[0 0 1512 2016] rois=[0 0 1512 2016]
rois=[0 0 1512 2016]
extent=[infinite][empty] extent=[infinite][empty]
extent=[-6 -6 1524 2028]
digest=893D3744D6DC6041 digest=491D0A9935DB1FC7
digest=10DBE3AD6B4EF2A0

4 6

let info = try? task.waitUntilCompleted()

kernel _cubicUpsample

(CIRenderTask) 0x0000000102806b20
3 5
premultiply
rois=[0 0 1512 2016]
extent=[infinite][empty]
digest=41DC5A181A04E2FD
provider BGRA8 32x1024
alpha_one edge_clamp
rois=[0 0 32 1024]
extent=[infinite][0 0 32 1024]
opaque
digest=4EC4D10594168D56
premultiply
rois=[0 0 1512 2016]
extent=[infinite][empty]
digest=41DC5A181A04E2FD
src
provider BGRA8 32x1024
scale=[0.380952 0.380952 0 0]
alpha_one edge_clamp
coefsLT1=[1.5 -2.5 0 1]
rois=[0 0 32 1024]
coefsLT2=[-0.5 2.5 -4 2]
extent=[infinite][0 0 32 1024]
rois=[0 0 1512 2016]
opaque
extent=[-6 -6 1524 2028]
digest=9EA3D447FA8130A0
digest=EE11208C4277FD65

else { linear_to_srgb
rois=[0 0 1512 2016]
extent=[infinite][empty]
digest=7085717B16282FC6
linear_to_srgb
rois=[0 0 1512 2016]
extent=[infinite][empty]
digest=7085717B16282FC6
affine
0 -1 576
-1 -0 768
rois=[-1 -1 578 770]
extent=[0 0 576 768]
opaque
digest=F26AE90AEAE2BD7E

// handle render failure affine

0 -0.5 1512
-0.5 -0 2016
rois=[0 0 1512 2016]
color_matrix_rrra
rois=[-1 -1 770 578]
extent=[0 0 768 576]
opaque
extent=[infinite][empty]
digest=B8397104A14159DD

}
digest=F1AC6EB48CC9C297

color_matrix_3x3
r=[1.22486 -0.225095 1.43051e-05]
crop [0 0 768 576]
g=[-0.0420312 1.04215 3.36338e-05]
rois=[-1 -1 770 578]
b=[-0.0196301 -0.0786319 1.09799]
extent=[0 0 768 576]
rois=[0 0 4032 3024]
opaque
extent=[infinite][empty]
digest=9310A62A244DBB44
opaque
digest=56878F059714A71E

curve
gamma=2.39999 swizzle_rrr1
a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0 rois=[0 0 768 576]
rois=[0 0 4032 3024] extent=[infinite][empty]
extent=[infinite][empty] opaque
opaque digest=FEEF5EFAAB178D1C
digest=1227B74C49153950

color_matrix_3x4
r=[1 0 1.40752]
g=[1 -0.345491 -0.716948] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
b=[1 1.77898 0] rois=[0 0 768 576]
bias=[-0.70652 0.533302 -0.892976] extent=[infinite][empty]
rois=[0 0 4032 3024] opaque
extent=[infinite][empty] digest=3A29A615F9F448A
opaque
digest=23C22371D291549D

IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp

rois=[0 0 4032 3024]
extent=[infinite][empty]
opaque
digest=33944A11D3550468
 -1 -0 768
rois=[0 0 1512 2016] rois=[0 0 1512 2016]
rois=[-1 -1 578 770]
extent=[infinite][empty] extent=[infinite][empty]
extent=[0 0 576 768]
digest=7085717B16282FC6 digest=7085717B16282FC6
opaque
digest=F26AE90AEAE2BD7E

affine
color_matrix_rrra
0 -0.5 1512
rois=[-1 -1 770 578]
-0.5 -0 2016
extent=[0 0 768 576]
rois=[0 0 1512 2016]
opaque
extent=[infinite][empty]
digest=B8397104A14159DD
digest=F1AC6EB48CC9C297

color_matrix_3x3
r=[1.22486 -0.225095 1.43051e-05]
crop [0 0 768 576]
g=[-0.0420312 1.04215 3.36338e-05]
rois=[-1 -1 770 578]
b=[-0.0196301 -0.0786319 1.09799]
extent=[0 0 768 576]
rois=[0 0 4032 3024]
opaque
extent=[infinite][empty]
digest=9310A62A244DBB44
opaque
digest=56878F059714A71E

curve
gamma=2.39999 swizzle_rrr1
a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0 rois=[0 0 768 576]
rois=[0 0 4032 3024] extent=[infinite][empty]
extent=[infinite][empty] opaque
opaque digest=FEEF5EFAAB178D1C
digest=1227B74C49153950

color_matrix_3x4
r=[1 0 1.40752]
g=[1 -0.345491 -0.716948] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
 -1 -0 768
color_matrix_rrra
rois=[0 0 1512 2016]
linear_to_srgb rois=[0 0 1512 2016]
linear_to_srgb
rois=[-1
rois=[-1 -1 578
-1 770 578]770]
extent=[infinite][empty]
rois=[0 0 1512 2016] extent=[infinite][empty]
rois=[0 0 1512 2016]
extent=[0
extent=[0 0 576
0 768 576]768]
digest=7085717B16282FC6
extent=[infinite][empty] digest=7085717B16282FC6
extent=[infinite][empty]
opaque
opaque
digest=99F3F681729FA5A4 digest=99F3F681729FA5A4
digest=F26AE90AEAE2BD7E
digest=F61974C92D5B0B9E

affine
affine
cropcolor_matrix_rrra
[0 0 768 576]
00 -0.5
-0.5 1512
1512
rois=[-1
rois=[-1 -1 770
-1 770 578]578]
-0.5 -0
-0.5 -0 2016
2016
extent=[0
extent=[0 0 768
0 768 576]576]
rois=[0 00 1512
rois=[0 1512 2016]
2016]
opaque
opaque
extent=[infinite][empty]
extent=[infinite][empty]
digest=B8397104A14159DD
digest=179BA22AFF932D96
digest=F1AC6EB48CC9C297
digest=7543B6B4D80E35E3

color_matrix_3x3
color_matrix_3x3
r=[1.22486 -0.225095
r=[1.22486 -0.225095 1.43051e-05]
1.43051e-05]
crop [0 0 768 576]
swizzle_rrr1
g=[-0.0420312 1.04215
g=[-0.0420312 1.04215 3.36338e-05]
3.36338e-05]
rois=[-1
rois=[0 0 768-1576]
770 578]
b=[-0.0196301 -0.0786319
b=[-0.0196301 -0.0786319 1.09799]
1.09799]
extent=[0 0 768 576]
extent=[infinite][empty]
rois=[0 00 4032
rois=[0 4032 3024]
3024]
opaque
opaque
extent=[infinite][empty]
extent=[infinite][empty]
digest=9310A62A244DBB44
digest=7476CC0097772826
opaque
opaque
digest=56878F059714A71E
digest=CFB3EC1C5985E7E0

curve
curve
gamma=2.39999
gamma=2.39999 IOSurface 0x0(0)swizzle_rrr1
seed:0 Nil alpha_one edge_clamp
a=0.947998 b=0.052002
a=0.947998 b=0.052002 c=0.0769958
c=0.0769958 d=0.0390015
d=0.0390015 e=0
e=0 f=0
f=0 rois=[0 0 768 576]
rois=[0 0 768 576]
rois=[0 00 4032
rois=[0 4032 3024]
3024] extent=[infinite][empty]
extent=[infinite][empty]
extent=[infinite][empty]
extent=[infinite][empty] opaque opaque
opaque
opaque digest=FEEF5EFAAB178D1C
digest=FF5B1FBE78FF1C9E
digest=1227B74C49153950
digest=6BDBF61A00035E59

color_matrix_3x4
color_matrix_3x4
r=[1 00 1.40752]
r=[1 1.40752]
g=[1 -0.345491
g=[1 -0.345491 -0.716948]
-0.716948] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
 -1 -0 768
rois=[0 0 1512 2016] rois=[0 0 1512 2016] opaque
rois=[-1 -1 578 770]
extent=[infinite][empty] extent=[infinite][empty] digest=F26AE90AEAE
extent=[0 0 576 768]
digest=7085717B16282FC6 digest=7085717B16282FC6
opaque
digest=F26AE90AEAE2BD7E

affine
affine
color_matrix_rrra
color_matrix_rrra
0 0-0.5
-0.5 1512
1512
rois=[-1 -1 770 578] rois=[-1 -1 770 57
-0.5
-0.5 -0-0 2016
2016
extent=[0 0 768 576]
rois=[0 0 1512 2016] extent=[0 0 768 57
rois=[0 0 1512 2016] opaque
extent=[infinite][empty] opaque
extent=[infinite][empty] digest=B8397104A14159DD
digest=F1AC6EB48CC9C297 digest=B8397104A14
digest=F1AC6EB48CC9C297

color_matrix_3x3
r=[1.22486 -0.225095 1.43051e-05]
crop [0 0 768 576]
g=[-0.0420312 1.04215 3.36338e-05]
color_matrix_3x3
b=[-0.0196301 -0.0786319 1.09799]
rois=[-1 -1 770 578]
extent=[0 0 768 576]
r=[1.22486
rois=[0 0 -0.225095
4032 3024] 1.43051e-05]
extent=[infinite][empty]
opaque crop [0 0 768 576]
g=[-0.0420312 1.04215 3.36338e-05] digest=9310A62A244DBB44
opaque rois=[-1 -1 770 57
b=[-0.0196301 -0.0786319 1.09799]
digest=56878F059714A71E extent=[0 0 768 57
rois=[0 0 4032 3024]
opaque
extent=[infinite][empty]
digest=9310A62A244
opaque
digest=56878F059714A71E
curve
gamma=2.39999 swizzle_rrr1
a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0 rois=[0 0 768 576]
rois=[0 0 4032 3024] extent=[infinite][empty]
extent=[infinite][empty] opaque
opaque digest=FEEF5EFAAB178D1C
digest=1227B74C49153950
curve
gamma=2.39999 swizzle_rrr1
a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0 rois=[0 0 768 576]
rois=[0 0 4032 3024] extent=[infinite][
extent=[infinite][empty]
color_matrix_3x4 opaque
r=[1 0 1.40752]
opaque digest=FEEF5EFAAB1
g=[1 -0.345491 -0.716948] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
optimized graph
render_to_surface
(metal context 1 frame 1)
format=BGRA8 roi=[0 0 1512 2016]

affine
1 0 0
0 -1 2016
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=114BF1F26BC2E310

clamp_to_alpha
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=7FF7DA85C4C272F

premultiply
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=C1C4C4605B7A67F8

linear_to_srgb
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=28A3941A3DEF8ED3

unpremultiply
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=6E8216B0766EE943
optimized graph
render_to_surface
(metal context 1 frame 1)
format=BGRA8 roi=[0 0 1512 2016]

affine
1 0 0
0 -1 2016
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=114BF1F26BC2E310

clamp_to_alpha
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=7FF7DA85C4C272F

premultiply
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=C1C4C4605B7A67F8

linear_to_srgb
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=28A3941A3DEF8ED3

unpremultiply
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=6E8216B0766EE943
CIRenderInfo Quick Look
// render output to an IOSurface
let dest = CIRenderDestination(ioSurface: surface);

let context = self.context;

guard
let task = try? context.startTask(toRender: output, to: dest),
let info = try? task.waitUntilCompleted()
else {
// handle render failure
}
CIRenderInfo Quick Look
// render output to an IOSurface
let dest = CIRenderDestination(ioSurface: surface);

let context = self.context;

guard
let task = try? context.startTask(toRender: output, to: dest),
let info = try? task.waitUntilCompleted()
else { (CIRenderInfo) 0x0000000100e082b0
// handle render failure
}
CIRenderInfo Quick Look
program graph
render_to_surface
(metal context 1 frame 1 tile 1)
format=BGRA8 roi=[0 0 1512 2016]

// render output to an IOSurface

program
affine [1 0 0 -1 0 2016]

let dest = CIRenderDestination(ioSurface: surface); clamp_to_alpha

premultiply
linear_to_srgb
unpremultiply
colorkernel _blendWithMask f b m
premultiply
srgb_to_linear
unpremultiply
colorkernel _colorcubeopaque_denorm im cube=(1) dims=[31 0.0322581 0.03125 0.000976562]
premultiply

let context = self.context;

linear_to_srgb
affine [0 -0.5 -0.5 -0 1512 2016]
color_matrix_3x3 r=[1.22486 -0.225095 1.43051e-05] g=[-0.0420312 1.04215 3.36338e-05] b=[-0.0196301 -0.0786319 1.09799]
curve gamma=2.39999 a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0
color_matrix_3x4 r=[1 0 1.40752] g=[1 -0.345491 -0.716948] b=[1 1.77898 0] bias=[-0.70652 0.533302 -0.892976]
(0)
(1)
premultiply
srgb_to_linear
unpremultiply
colorkernel _colorcubeopaque_denorm im cube=(2) dims=[31 0.0322581 0.03125 0.000976562]

guard
premultiply
linear_to_srgb
affine [0 -0.5 -0.5 -0 1512 2016]
color_matrix_3x3 r=[1.22486 -0.225095 1.43051e-05] g=[-0.0420312 1.04215 3.36338e-05] b=[-0.0196301 -0.0786319 1.09799]
curve gamma=2.39999 a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0
color_matrix_3x4 r=[1 0 1.40752] g=[1 -0.345491 -0.716948] b=[1 1.77898 0] bias=[-0.70652 0.533302 -0.892976]

let task = try? context.startTask(toRender: output, to: dest),

(0)
(2)
premultiply
colorkernel _colorClamp c lo=[0 0 0 0] hi=[1 1 1 1]
unpremultiply

let info = try? task.waitUntilCompleted()

color_matrix_3x4 r=[2 0 0] g=[0 2 0] b=[0 0 2] bias=[-0.5 -0.5 -0.5]
(3)
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=114BF1F26BC2E310

(CIRenderInfo) 0x0000000100e082b0
time=15.05994 ms

else {
3
1 2

// handle render failure

0
program format=RGBAh
provider BGRA8 32x1024 provider BGRA8 32x1024
kernel _cubicUpsample src scale=[0.380952 0.380952 0 0] coefsLT1=[1.5 -2.5 0 1] coefsLT2=[-0.5 2.5 -4 2] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
alpha_one edge_clamp alpha_one edge_clamp
affine [0 -1 -1 -0 576 768] rois=[0 0 4032 3024]
rois=[0 0 32 1024] rois=[0 0 32 1024]
rois=[0 0 1512 2016] extent=[infinite][empty]
extent=[infinite][0 0 32 1024] extent=[infinite][0 0 32 1024]
extent=[-6 -6 1524 2028] opaque

}
opaque opaque
digest=EE11208C4277FD65 digest=33944A11D3550468
digest=4EC4D10594168D56 digest=9EA3D447FA8130A0
time=1.52849 ms

program format=RGBAh
color_matrix_rrra
crop [0 0 768 576]
swizzle_rrr1
rois=[-1 -1 770 578]
extent=[0 0 768 576]
opaque
digest=B8397104A14159DD
time=0.06279 ms

IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp

rois=[0 0 768 576]
extent=[infinite][empty]
opaque
digest=3A29A615F9F448A
 srgb_to_linear
unpremultiply
colorkernel _colorcubeopaque_denorm im cube=(2) dims=[31 0.0322581 0.03125 0.000976562]
premultiply
linear_to_srgb
affine [0 -0.5 -0.5 -0 1512 2016]
color_matrix_3x3 r=[1.22486 -0.225095 1.43051e-05] g=[-0.0420312 1.04215 3.36338e-05] b=[-0.0196301 -0.0786319 1.09799]
curve gamma=2.39999 a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0
color_matrix_3x4 r=[1 0 1.40752] g=[1 -0.345491 -0.716948] b=[1 1.77898 0] bias=[-0.70652 0.533302 -0.892976]
(0)
(2)
premultiply
colorkernel _colorClamp c lo=[0 0 0 0] hi=[1 1 1 1]
unpremultiply
color_matrix_3x4 r=[2 0 0] g=[0 2 0] b=[0 0 2] bias=[-0.5 -0.5 -0.5]
(3)
rois=[0 0 1512 2016]
extent=[0 0 1512 2016]
digest=114BF1F26BC2E310
time=15.05994 ms

3
1 2
0
program format=RGBAh
provider BGRA8 32x1024 provider BGRA8 32x1024
kernel _cubicUpsample src scale=[0.380952 0.380952 0 0] coefsLT1=[1.5 -2.5 0 1] coefsLT2=[-0.5 2.5 -4 2] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
alpha_one edge_clamp alpha_one edge_clamp
affine [0 -1 -1 -0 576 768] rois=[0 0 4032 3024]
rois=[0 0 32 1024] rois=[0 0 32 1024]
rois=[0 0 1512 2016] extent=[infinite][empty]
extent=[infinite][0 0 32 1024] extent=[infinite][0 0 32 1024]
extent=[-6 -6 1524 2028] opaque
opaque opaque
digest=EE11208C4277FD65 digest=33944A11D3550468
digest=4EC4D10594168D56 digest=9EA3D447FA8130A0
time=1.52849 ms

program format=RGBAh
color_matrix_rrra
crop [0 0 768 576]
swizzle_rrr1
rois=[-1 -1 770 578]
extent=[0 0 768 576]
opaque
digest=B8397104A14159DD
time=0.06279 ms

IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp

rois=[0 0 768 576]
extent=[infinite][empty]
opaque
digest=3A29A615F9F448A
 srgb_to_linear colorkernel _colorClamp c lo=[0 0 0 0] hi=[1 1 1 1]
unpremultiply
unpremultiply
colorkernel _colorcubeopaque_denorm im cube=(2) dims=[31 0.0322581 0.03125 0.000976562]
premultiply color_matrix_3x4 r=[2 0 0] g=[0 2 0] b=[0 0 2] bias=[-0.5 -0.5 -0.5]
linear_to_srgb (3)
affine [0 -0.5 -0.5 -0 1512 2016]
rois=[0 0color_matrix_3x3
1512 2016] r=[1.22486 -0.225095 1.43051e-05] g=[-0.0420312 1.04215 3.36338e-05] b=[-0.0196301 -0.0786319 1.09799]
extent=[0 0 1512
curve 2016] a=0.947998 b=0.052002 c=0.0769958 d=0.0390015 e=0 f=0
gamma=2.39999
color_matrix_3x4 r=[1 0 1.40752] g=[1 -0.345491 -0.716948] b=[1 1.77898 0] bias=[-0.70652 0.533302 -0.892976]
digest=114BF1F26BC2E310
(0)
time=15.05994
(2) ms
premultiply
colorkernel _colorClamp c lo=[0 0 0 0] hi=[1 1 1 1]
unpremultiply
color_matrix_3x4 r=[2 0 0] g=[0 2 0] b=[0 0 2] bias=[-0.5 -0.5 -0.5]
(3)
rois=[0 0 1512 2016]
3 extent=[0 0 1512 2016]
digest=114BF1F26BC2E310
1
0
m format=RGBAh time=15.05994 ms
provider BGRA8 32x1024
_cubicUpsample src scale=[0.380952 0.380952 0 0] coefsLT1=[1.5 -2.5 0 1] coefsLT2=[-0.5 2.5 -4 2] IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
alpha_one edge_clamp
ne [0 -1 -1 -0 576 768] rois=[0 0 4032 3024]
rois=[0 0 32 1024]
0 0 1512 2016] 3
extent=[infinite][empty]
1 extent=[infinite][0
2 0 32
=[-6 -6 1524 2028] 0opaque
program format=RGBAh opaque
=EE11208C4277FD65
kernel _cubicUpsample src scale=[0.380952 0.380952 0 0] coefsLT1=[1.5 -2.5 0 1] coefsLT2=[-0.5 2.5 -4 2] digest=33944A11D3550468
IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
provider BGRA8 32x1024 provider BGRA8 32x1024
alpha_one edge_clamp alpha_one digest=4EC4D10594168D56
edge_clamp
.52849 ms affine [0 -1 -1 -0 576 768] rois=[0 0 4032 3024]
rois=[0 0 32 1024] rois=[0 0 32 1024]
rois=[0 0 1512 2016] extent=[infinite][empty]
extent=[infinite][0 0 32 1024] extent=[infinite][0 0 32 1024]
extent=[-6 -6 1524 2028] opaque
opaque opaque
digest=EE11208C4277FD65 digest=33944A11D3550468
digest=4EC4D10594168D56 digest=9EA3D447FA8130A0
time=1.52849 ms

program format=RGBAh
color_matrix_rrra
program format=RGBAh
color_matrix_rrra
crop [0 0 768 576]
crop [0 0 768 576]
swizzle_rrr1
swizzle_rrr1
rois=[-1
rois=[-1-1 770578]
-1 770 578]
extent=[0 0 768 576]
extent=[0 0 768 576]
opaque
opaque
digest=B8397104A14159DD
time=0.06279 ms
digest=B8397104A14159DD
time=0.06279 ms

IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp

rois=[0 0 768 576]
extent=[infinite][empty]
opaque
IOSurface 0x0(0) seed:0 Nil alpha_one edge_clamp
digest=3A29A615F9F448A

rois=[0 0 768 576]

extent=[infinite][empty]
opaque
CIBarcodeDescriptor API
•
Framework Barcode Support
Several frameworks support various barcodes types

Code128

Aztec
QRCode

PDF417
Framework Barcode Support
Several frameworks support various barcodes types

AVFoundation.framework Vision.framework CoreImage.framework

Detection of barcodes
 Detection of barcodes


Rendering barcodes
during capture after capture
Framework Barcode Support
Several frameworks support various barcodes types

AVFoundation.framework Vision.framework CoreImage.framework

Detection of barcodes
 Detection of barcodes


Rendering barcodes
during capture after capture

CIBarcodeDescriptor
CIBarcodeDescriptor Objects Contain

The errorCorrectedPayload of the barcode

CIBarcodeDescriptor Objects Contain

The errorCorrectedPayload of the barcode

Additional code-specific properties such as

CIBarcodeDescriptor Objects Contain

The errorCorrectedPayload of the barcode

Additional code-specific properties such as

• Aztec’s layerCount
CIBarcodeDescriptor Objects Contain

The errorCorrectedPayload of the barcode

Additional code-specific properties such as

• Aztec’s layerCount

• QRCode’s maskPattern
// Get a CIBarcodeDescriptor from AVFoundation.framework

class MyMetadataOutputObjectsDelegate: NSObject, AVCaptureMetadataOutputObjectsDelegate

{
func metadataOutput(_ output: AVCaptureMetadataOutput,
didOutput metadataObjects: [AVMetadataObject],
from connection: AVCaptureConnection) {
if let mrc = metadataObjects.first as? AVMetadataMachineReadableCodeObject,
let descriptor = mrc.descriptor {
print(descriptor)
}
}
}
// Get a CIBarcodeDescriptor from AVFoundation.framework

class MyMetadataOutputObjectsDelegate: NSObject, AVCaptureMetadataOutputObjectsDelegate

{
func metadataOutput(_ output: AVCaptureMetadataOutput,
didOutput metadataObjects: [AVMetadataObject],
from connection: AVCaptureConnection) {
if let mrc = metadataObjects.first as? AVMetadataMachineReadableCodeObject,
let descriptor = mrc.descriptor {
print(descriptor)
}
}
}
// Detect a CIBarcodeDescriptor using Vision.framework

func descriptorFromImage(_ image: CIImage) -> CIBarcodeDescriptor?

{
// Create the request and request handler
let requestHandler = VNImageRequestHandler(ciImage: image, options: [:])
let request = VNDetectBarcodesRequest();

// Send the request to the handler

try? requestHandler.perform([request])

// Get the observation

let firstResult = request.results?.first
return firstResult?.barcodeDescriptor
}
// Detect a CIBarcodeDescriptor using Vision.framework

func descriptorFromImage(_ image: CIImage) -> CIBarcodeDescriptor?

{
// Create the request and request handler
let requestHandler = VNImageRequestHandler(ciImage: image, options: [:])
let request = VNDetectBarcodesRequest();

// Send the request to the handler

try? requestHandler.perform([request])

// Get the observation

let firstResult = request.results?.first
return firstResult?.barcodeDescriptor
}
// Detect a CIBarcodeDescriptor using Vision.framework

func descriptorFromImage(_ image: CIImage) -> CIBarcodeDescriptor?

{
// Create the request and request handler
let requestHandler = VNImageRequestHandler(ciImage: image, options: [:])
let request = VNDetectBarcodesRequest();

// Send the request to the handler

try? requestHandler.perform([request])

// Get the observation

let firstResult = request.results?.first
return firstResult?.barcodeDescriptor
}
// Detect a CIBarcodeDescriptor using Vision.framework

func descriptorFromImage(_ image: CIImage) -> CIBarcodeDescriptor?

{
// Create the request and request handler
let requestHandler = VNImageRequestHandler(ciImage: image, options: [:])
let request = VNDetectBarcodesRequest();

// Send the request to the handler

try? requestHandler.perform([request])

// Get the observation

let firstResult = request.results?.first
return firstResult?.barcodeDescriptor
}
// Create an image for a CIBarcodeDescriptor using CoreImage.framework

func imageFromBarcodeCodeDescriptor(_ descriptor: CIBarcodeDescriptor) -> CIImage?

{
return CIFilter(name: "CIBarcodeGenerator",
withInputParameters: ["inputBarcodeDescriptor" : descriptor])
?.outputImage
}
// Create an image for a CIBarcodeDescriptor using CoreImage.framework

func imageFromBarcodeCodeDescriptor(_ descriptor: CIBarcodeDescriptor) -> CIImage?

{
return CIFilter(name: "CIBarcodeGenerator",
withInputParameters: ["inputBarcodeDescriptor" : descriptor])
?.outputImage
}
// Create an image for a CIBarcodeDescriptor using CoreImage.framework

func imageFromBarcodeCodeDescriptor(_ descriptor: CIBarcodeDescriptor) -> CIImage?

{
return CIFilter(name: "CIBarcodeGenerator",
withInputParameters: ["inputBarcodeDescriptor" : descriptor])
?.outputImage
}
// Create an image for a CIBarcodeDescriptor using CoreImage.framework

func imageFromBarcodeCodeDescriptor(_ descriptor: CIBarcodeDescriptor) -> CIImage?

{
return CIFilter(name: "CIBarcodeGenerator",
withInputParameters: ["inputBarcodeDescriptor" : descriptor])
?.outputImage
}
Demo
•

Detection and Re-Generation of a QRCode

Using Core Image with Vision
•
Using Core Image with Vision
Using Core Image with Vision

Core Image can prepare images before being passed to Vision

• For example: Cropping, Orienting, Converting to grayscale
Using Core Image with Vision

Core Image can prepare images before being passed to Vision

• For example: Cropping, Orienting, Converting to grayscale

Vision can gather information before processing with Core Image

• For example: Feature detection can guide which CIFilters to apply
Photo from Video with Removal of Unwanted Objects

AVFoundation: Get the frames out of a short video

Photo from Video with Removal of Unwanted Objects

AVFoundation: Get the frames out of a short video

Photo from Video with Removal of Unwanted Objects

Vision: Determine homography matrices to align frames

Photo from Video with Removal of Unwanted Objects

Vision: Determine homography matrices to align frames

Photo from Video with Removal of Unwanted Objects

Core Image: Align each video frame

Photo from Video with Removal of Unwanted Objects

Core Image: Make a photo from the median of the aligned frames
Photo from Video with Removal of Unwanted Objects

Core Image: Make a photo from the median of the aligned frames
Photo from Video with Removal of Unwanted Objects

Core Image: Make a photo from the median of the aligned frames
Photo from Video with Removal of Unwanted Objects

Core Image: Make a photo from the median of the aligned frames
// Use Vision to find homographic registration and pass it to CI

func homographicTransform(from image: CIImage, to reference: CIImage) -> matrix_float3x3? {

// Create the request and request handler
let request = VNHomographicImageRegistrationRequest(targetedCIImage: image);
let requestHandler = VNImageRequestHandler(ciImage: reference, options: [:]);

// Send the request to the handler

try? requestHandler.perform([request]);

// Get the observation

guard let results = request.results,
let observation = results.first as? VNImageHomographicAlignmentObservation
else {
return nil
}
return observation.warpTransform
}
// Use Vision to find homographic registration and pass it to CI

func homographicTransform(from image: CIImage, to reference: CIImage) -> matrix_float3x3? {

// Create the request and request handler
let request = VNHomographicImageRegistrationRequest(targetedCIImage: image);
let requestHandler = VNImageRequestHandler(ciImage: reference, options: [:]);

// Send the request to the handler

try? requestHandler.perform([request]);

// Get the observation

guard let results = request.results,
let observation = results.first as? VNImageHomographicAlignmentObservation
else {
return nil
}
return observation.warpTransform
}
// Use Vision to find homographic registration and pass it to CI

func homographicTransform(from image: CIImage, to reference: CIImage) -> matrix_float3x3? {

// Create the request and request handler
let request = VNHomographicImageRegistrationRequest(targetedCIImage: image);
let requestHandler = VNImageRequestHandler(ciImage: reference, options: [:]);

// Send the request to the handler

try? requestHandler.perform([request]);

// Get the observation

guard let results = request.results,
let observation = results.first as? VNImageHomographicAlignmentObservation
else {
return nil
}
return observation.warpTransform
}
// Use Vision to find homographic registration and pass it to CI

func homographicTransform(from image: CIImage, to reference: CIImage) -> matrix_float3x3? {

// Create the request and request handler
let request = VNHomographicImageRegistrationRequest(targetedCIImage: image);
let requestHandler = VNImageRequestHandler(ciImage: reference, options: [:]);

// Send the request to the handler

try? requestHandler.perform([request]);

// Get the observation

guard let results = request.results,
let observation = results.first as? VNImageHomographicAlignmentObservation
else {
return nil
}
return observation.warpTransform
}
// Use Vision to find homographic registration and pass it to CI

func homographicTransform(from image: CIImage, to reference: CIImage) -> matrix_float3x3? {

// Create the request and request handler
let request = VNHomographicImageRegistrationRequest(targetedCIImage: image);
let requestHandler = VNImageRequestHandler(ciImage: reference, options: [:]);

// Send the request to the handler

try? requestHandler.perform([request]);

// Get the observation

guard let results = request.results,
let observation = results.first as? VNImageHomographicAlignmentObservation
else {
return nil
}
return observation.warpTransform
}
// Core Image Metal kernel to apply a homography matrix

float2 warpHomography(float3x3 h, destination dest)

{
float3 homogeneousDestCoord = float3(dest.coord(), 1.0);
float3 homogeneousSrcCoord = h * homogeneousDestCoord;
float2 srcCoord = homogeneousSrcCoord.xy / max(homogeneousSrcCoord.z, 0.000001);
return srcCoord;
}
// Core Image Metal kernel to apply a homography matrix

float2 warpHomography(float3x3 h, destination dest)

{
float3 homogeneousDestCoord = float3(dest.coord(), 1.0);
float3 homogeneousSrcCoord = h * homogeneousDestCoord;
float2 srcCoord = homogeneousSrcCoord.xy / max(homogeneousSrcCoord.z, 0.000001);
return srcCoord;
}
// Core Image Metal kernel to apply a homography matrix

float2 warpHomography(float3x3 h, destination dest)

{
float3 homogeneousDestCoord = float3(dest.coord(), 1.0);
float3 homogeneousSrcCoord = h * homogeneousDestCoord;
float2 srcCoord = homogeneousSrcCoord.xy / max(homogeneousSrcCoord.z, 0.000001);
return srcCoord;
}
// Core Image Metal kernel to apply a homography matrix

float2 warpHomography(float3x3 h, destination dest)

{
float3 homogeneousDestCoord = float3(dest.coord(), 1.0);
float3 homogeneousSrcCoord = h * homogeneousDestCoord;
float2 srcCoord = homogeneousSrcCoord.xy / max(homogeneousSrcCoord.z, 0.000001);
return srcCoord;
}
// Core Image Metal kernel to return the median of 5 images

inline void swap(thread float4 &a, thread float4 &b) {

float4 tmp = a; a = min(a,b); b = max(tmp, b); // swap sort of two elements
}

float4 medianReduction5(sample_t v0, sample_t v1, sample_t v2, sample_t v3, sample_t v4)
{
// using a Bose-Nelson sorting network
swap(v0, v1); swap(v3, v4); swap(v2, v4); swap(v2, v3); swap(v0, v3);
swap(v0, v2); swap(v1, v4); swap(v1, v3); swap(v1, v2);
return v2;
}
// Core Image Metal kernel to return the median of 5 images

inline void swap(thread float4 &a, thread float4 &b) {

float4 tmp = a; a = min(a,b); b = max(tmp, b); // swap sort of two elements
}

float4 medianReduction5(sample_t v0, sample_t v1, sample_t v2, sample_t v3, sample_t v4)
{
// using a Bose-Nelson sorting network
swap(v0, v1); swap(v3, v4); swap(v2, v4); swap(v2, v3); swap(v0, v3);
swap(v0, v2); swap(v1, v4); swap(v1, v3); swap(v1, v2);
return v2;
}
// Core Image Metal kernel to return the median of 5 images

inline void swap(thread float4 &a, thread float4 &b) {

float4 tmp = a; a = min(a,b); b = max(tmp, b); // swap sort of two elements
}

float4 medianReduction5(sample_t v0, sample_t v1, sample_t v2, sample_t v3, sample_t v4)
{
// using a Bose-Nelson sorting network
swap(v0, v1); swap(v3, v4); swap(v2, v4); swap(v2, v3); swap(v0, v3);
swap(v0, v2); swap(v1, v4); swap(v1, v3); swap(v1, v2);
return v2;
}
Demo
•

Photo from Video with Removal of Unwanted Objects

Sky Gao, Image Engineer

Summary

Performance Information Functionality

Write CIKernels in Metal CIRenderInfo API New Filters

CIRenderDestination API Xcode Quick Looks Barcode Support
Depth Support
 More Information
https://developer.apple.com/wwdc17/510
Related Sessions

Image Editing with Depth WWDC 2017

What's New in Photos APIs WWDC 2017

Vision Framework: Building on Core ML WWDC 2017

Capturing Depth in iPhone Photography WWDC 2017

Labs

Photos Editing and Core Image Lab Technology Lab F Thu 3:10PM–6:00PM

Photos Depth and Capture Lab Technology Lab A Thu 3:10PM–6:00PM

Photos Depth and Capture Lab Technology Lab F Fri 1:50PM–4:00PM