Sky replacement tools are extremely popular in the current crop of photp-editing apps. Mobile apps such as Lighttricks’ Lightleap help you put up a new sky on your picture quickly. While for a more refined option, you may want to turn to Adobe Photoshop’s recently-released AI-powered feature.

However, there aren’t many commercially available tools for such video effects. A new open-source tool takes a crack at replacing the sky with AR effects in clips. The unique thing about this AI-powered tool is that it can also augment objects in the sky and adjust the footage as if it was actually present during the shooting.

The model is independent of the shooting source. So, you can use footage captured even from your phone. Plus, you can use a photo or a video in place of the sky. It can even create a Skybox effect to make your video look larger than life. Imagine sticking up a giant spaceship in a clip of your road trip!

There are a couple of limitations to this model. First, it’s trained only for the daytime sky. So, if your video was shot at night, the tool will not work effectively. Second, if the sky was clear with no object in sight, it might replace some of the foregrounds as well. Not an ideal result for video editors.

However, this is a great starting point for potential mainstream tools to experiment with and integrate into their products.

You can read the full paper titled “Castle in the Sky” here. You can even give it a spin on your Windows laptop by following this tutorial.

Published November 16, 2020 — 07:31 UTC