So after another couple hundred hours of work I'm putting this project on hold on my end as I wait for technology to improve and become more accessible. I've learned a lot though that I want to share with the community so if anyone else has ideas, I'm willing to listen. The first episode I released was using only Adobe Premiere. I've continued playing around with Adobe Premiere and After Effects using AE's many masking functions but they don't work well for this purpose at scale (90 minutes of material). OpenCV which is a python image-processing library was an alternative which I still think has promise but I've implemented all of the ideas I've had and none of them have lead to what I'd consider an acceptable solution.
What I have learned, some of which I mentioned prior:
-The MST3K DVD encodes can be substantively improved using the QTGMC deinterlacing filter. It's by-far ahead of other filters and is an important first step in getting the best shadowrama as well. Here is a zip file with everything someone would need in order to set up the process and begin encoding (there are a bunch of .dll's for AVIsynth plugins so it might be flagged when you download it):
https://mega.nz/file/c6wgEYzT#Xj73BQx7in...S0dwVgvYUs
-The next step prior to isolation is up-scaling the shadowrama, I ran a lot of tests trying things in a different order. Originally I had used the xBR algorithm which had proven better than several others. This time I tested 6 different DNN AI-upscaling algorithms as well.
FSRCNN was much better than the xBR but the proprietary software
Topaz Labs Video Enchance AI actually performed far and away the best. Note: this is the best for the
shadowrama itself, it really messes up human skin and eyes, giving it a weird effect. So I was planning on using FSRCNN for the host segments and Topaz Video to upscale the shadowrama. This is actually a huge improvement over xBR and it really looks like the shadowrama was filmed in 4k/HD, it's incredible.
-The real problem is with isolating the shadowrama. Thresholding alone isn't enough because of the surrounding scenery and inconsistency, I tried adaptive thresholding including creating my own version of the
Otsu method (I read a dozen or so academic papers on computer vision processing throughout this whole thing and luckily I have mathematics background to make sense of some of it). I also got a hold of
Silhouette 2020 which is an "oscar-winning" tool used by big-budget Hollywood studios for this sort of thing but that's not something a hobbyist like myself could really learn in a reasonable time (especially as they also make money by selling you training courses and I couldn't find those).
Some hope though: I actually think that
Mask R-CNN which is a Deep Learning Neural Network approach to semantic segmentation of objects would work but it would require training a specific model for this task which requires annotating images which is an incredibly manual and time-consuming process. I trained a model based off of 3 images I annotated but it's obviously not enough, and I didn't want to spend 200+ incredibly tedious hours annotating a thousand images for my machine learning model and find out that it still wouldn't work as well as I want. Topaz Labs also has a "
Mask AI" program that does what we want, but only for photos, not videos and there's no batch processing/it has some upfront manual effort. If they made a version of it for videos (and it's only a matter of time they or someone else does), then that's going to be the best solution. So right now, I think I'm going to play the waiting game for technology to come to improve as well as for it to become more accessible. The fun part of this whole project for me was the learning aspect and challenge and I don't want to spend a lot of time creating new episodes based on an approach which produces something I deem to be sub-standard.
If anyone wants to play around with some of my OpenCV code with a test clip to see it in action, you can download it here:
https://mega.nz/file/pqo0WYoS#2ESZkdUHbl...3tAoBP3Wps
I'd like to share the Adobe Premiere project, but I'm not sure the best way to do that as all of the working files add up to 24GB, but here are all the project files beyond the videos which should give an idea of the effects and methods from my first attempt, also if you think you've got skills, PM me and we'll figure out how to get you the complete project file:
https://mega.nz/file/Qig2BILS#NqLvUbyDPQ...cFA9fb5Se4
One funny note about this whole project: the holes in Crow's head make this whole damn thing 100x harder.
