I'm having similar issues, bought an EVGA GTX 680 for my MAC Pro 2010 (Intel 2x3.46GHz, 64GB DDR3, originally ATI 5770) to be able to render in octane, after effects and premiere, I setup a very simple scene (cube, sphere, cylinder and 3 lights) and everytime I hit render the MAC just reboots straight away, I don't even get a error log or something.
I'm wondering if the GPU is faulty, but to be honest, using the computer normally doesn’t seems to have any problems, browsing internet, any office applications and even some graphic design APP’s such as Indesign or Photoshop, including some gaming (Counter Strike GO, Diablo 3, World of Warcraft) all worked completely fine, the problem comes using the GPU for rendering with Premiere PRO, After Effects, or Cinema 4D using Octane.
All the times I tried any of these with the GTX680 the computer just turned off after a few seconds of render. I also have a PC with EVGA GTX 1080Ti which I tested on the MAC Pro, I had to install the NVIDIA web graphics to get the 1080 running on High-Sierra and tried the same APPs, all worked fine. I did also some tests with the GTX680 and web graphics but the same problem occurs. I did check about the compatibility between the GTX 680 and seems that all app's including octane are fully compatible.
Except from octane which needs CUDA, I tested the same after effects and premiere renders with the ATI 5770 and all also worked fine, that's where my questions come, is the EVGA GTX680 not strong enough to run octane? if so, why would be recommended on the octane and NVIDIA website?
or this may be a faulty GPU as the same crash/re-boot happens when intensely using the GPU?
Sorry if this is not the correct place to pos but as I saw a similar problem with similar GPU...
Thanks in advance,