Jump to content
Sign in to follow this  
NWoolridge

CV-AR: Easy facial performance capture

Recommended Posts

Cross- post from CGTalk:

 

Hi all,

 

I was lucky enough to be a beta tester for a cool new plugin from Cineversity: CV-AR.

 

CV-AR has two components: an iOS app that works with the TrueDepth Camera of the iPhone X to do facial motion capture; and a C4D plugin that takes that data and makes it available within C4D. Donovan Keith has an intro tutorial up on Cineversity at:

 

https://www.cineversity.com/vidplaylist/cv-ar

 

The free iOS app is available at:

 

https://itunes.apple.com/ca/app/cv-ar/id1378696551?mt=8

 

The iOS capture tool leverages Apple's ARkit to provide realtime capture of 51 facial movements, as well as eye orientation and head position.

 

The plugin is a blast to work with. The initial textured mask you get lets you know how well the performance was captured, but that's just the beginning. You can then use the blendshape strength data streamed from the capture object to drive your own rigs. I think it has real utility for animators, both as a way to provide reference for hand-tuned animation, and as a way to speed a more automated animation workflow.

 

Here is a sample movie with the capture object mask on the left and my rig driven by that data on the right. At this point I hadn't quite nailed some of the morph target sculpting, and the head movement needs adjustment. Let me know if you have any questions about how this works...

 

https://www.dropbox.com/s/aw5dev1iu...hapes2.mp4?dl=0

 

Props to Kent Barber for developing the plugin, and to Rick Barrett and the folks at Cineversity.

Share this post


Link to post
Share on other sites

Is planed support for android phones?

Share this post


Link to post
Share on other sites
  • Topic Author
  • 2 hours ago, bezo said:

    Is planed support for android phones?

    I’m just a beta tester, and not privy to the developers plans, so take this with a grain of salt, but: it may be a while, unfortunately. This plugin relies on the work Apple has done in ARKit to allow for reliable mobile face tracking, and in the 3D sensing hardware of the iPhone X (it is rumored that the TrueDepth camera will be spreading to other iOS devices in the future).

     

    Google is responding to ARKit with ARCore in Android, but it is not as mature, and does not have facial tracking. The earlier google AR effort, Tango, relies on specialized hardware, which might have been able to do face tracking, but they seem to be focusing on ARCore now. 

    Share this post


    Link to post
    Share on other sites

    Nick is spot on.

     

    I'm an Android user and would love to have an Android version, but the hardware and API aren't there yet. CV-AR makes use of the TrueDepth camera that's unique to the iPhoneX and the same API that powers their Animoji feature.

     

    So for now, it's limited to the iPhoneX. But we're excited about the possibilities of both iOS ARKit and Android's ARCore (as well as Snapchat's Lens Studio and Facebook's ARStudio), and hope to have more plugins and tutorials in the future that tie in to all the emerging augmented reality technologies.

     

     

    Share this post


    Link to post
    Share on other sites

    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.

    Guest
    Reply to this topic...

    ×   Pasted as rich text.   Paste as plain text instead

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    Sign in to follow this  

    • Recently Browsing   0 members

      No registered users viewing this page.

    ×
    ×
    • Create New...