A beautiful article about "CartesianIndices" in Julia

[dmm]

julialang.org/blog/2016/02/iteration/

(Caveats and remarks: This article is written for mutable arrays. We do often require immutable arrays these days.

Some of the links to Julia documentation are out of date, see docs.julialang.org/en/v1/manual/arrays/ instead.

X... construction typically "splats" elements of tuple or array X, see the second half of docs.julialang.org/en/v1/manual/functions/#Varargs-Functions section for explanation and examples, starting from the words 'On the flip side, it is often handy to "splat" the values contained in an iterable collection into a function call as individual arguments'; this construction works everywhere, not just in function calls.)

Comments

[dmm]

I decided I wanted to understand the internals of "imresize", and so I started reading

https://github.com/JuliaImages/ImageTransformations.jl/blob/master/src/ImageTransformations.jl

https://github.com/JuliaImages/ImageTransformations.jl/blob/master/src/autorange.jl

https://github.com/JuliaImages/ImageTransformations.jl/blob/master/src/resizing.jl

and I wanted to understand CartesianIndices which are used there. That's how I found that article.

[dmm]

Why on Earth do I want to understand the internals of "imresize"?

I am using a "warp" technique for wave effects in images, where I change the indices in a discrete way. e.g.

function warp3(x, y, knob) # 0:31
    clip(n, nsize) = max(min(nsize, n), 1)
    (x0, y0) = (clip(convert(Int, floor(x + 10 * ((knob - 16) / 16) * sin(y/4))), xsize), y) # normalize to get +/-        
    img[x0, y0]
end

All this needs to be refactored anyway:

https://github.com/anhinga/julia-notebooks/blob/main/grimoire-team/design-notes.md

But one thing I have noticed is that this cannot be meaningfully differentiated with respect to parameters such as knob. And what if I want to do some gradient descent with respect to knob?

Then I should do better than this. It does not have to be completely smooth, it can have breaks in the derivative, like in ReLU. But it should be a continuous function from knob.

[dmm]

And what I did learn about "imresize" is that it uses interpolations:

http://juliamath.github.io/Interpolations.jl/latest/interpolations/

https://github.com/JuliaMath/Interpolations.jl

[dmm]

:-) Of course, what we do, the "backward mode warping" is also extensively covered by that semi-documented package:

https://github.com/JuliaImages/ImageTransformations.jl/blob/master/src/warp.jl

https://github.com/JuliaImages/ImageTransformations.jl/blob/master/src/warpedview.jl

https://github.com/JuliaImages/ImageTransformations.jl/blob/master/src/invwarpedview.jl

I should at least try to read it (although I am a bit unsure about complexity; I tend to think about this rather simple-mindedly; and also - would Zygote take gradients through something like this? I am still unsure about actually calling these functions rather than implementing version(s) I have in mind).

(The official documentation to all this is absolutely minimal, namely

https://juliaimages.org/stable/pkgs/transformations/

but the functions are well-commented. Их схема более богатая; они там всякие соображения уже включили, которые накапливаются по мере того, как люди работают с этой схемой; в принципе, лучше бы пользоваться ей, если это не слишком сложно, и если оно не перестанет дифференцироваться из-за этого (e.g. we know that Zygote gradient-computing system enforces immutable arrays).

Но, с другой стороны, у них эти преобразования, в основном, линейные, и они используют этот сомнительный inv, а мы любим warp, который совсем не линеен и далеко не всегда обязан быть обратимым (хотя у нас необратимость, в основном, возникает из-за того, что мы отображаем целочисленный grid в целочисленный grid - как раз то, от чего мы хотим избавиться; но inv, всё равно, вряд ли будет работать).)

[dmm]

The next step: see how Grimoire warps will work will the official LinearInterpolation.

Will it be visually OK, etc...

[dmm]

The first refactor is done:

https://github.com/anhinga/julia-notebooks/tree/main/grimoire-team/refactor

Now we should start checking whether this works with Julia Flux and Zygote.

[dmm]

No, I am having problem with Flux/Zygote. I hoped this would work out of the box, but it will be necessary to debug things:

https://github.com/anhinga/julia-notebooks/tree/main/transition-to-flux

Other than that, the progress in recent days with this project was very nice.

[dmm]

Flux/Zygote works! Yay!

(But now it gets caught in a local minimum; anyway, now we can start thinking about the essence of the problem, and not about mere technical difficulties, like how to take derivative and such.)

[dmm]

Works now, and produces interesting results:

https://github.com/anhinga/julia-notebooks/tree/main/flux-may-2021