Jump to content

Archived

This topic is now archived and is closed to further replies.

Guest BTmonocle

Team Render woes / bandwidth issues

Recommended Posts

Guest BTmonocle
  • Topic Author
  • Hello everyone.
    New Cafe member here. Really hoping someone with a bigger brain  than my own can help me solve this issue. I've been talking with support at UK MAXON all morning who've been great, but it seems that while they're able to replicate the issue their end - it seems to fall outside of the scope of their support as it's probably network related.

    Long story short, we have a five machines here which I would love to utilize properly for a team render setup. Mainly i7 cpus and one 24 core Xeon workstation.

    Problem that I'm having is that it seems as soon as we get past two nodes in the set-up, each node that is added after that seems to depreciate massively in processing power contributed to the final render times. Which gets worse each time we add a node.

    I've attached some grabs of some tests I've been doing, rather than typing it all out. Appreciate there are a lot of variables with network rendering, but in my mind - with all the processing power we should be seeing bigger improvements to the render times. I'm using the grapes scene from the Studio folder as an example. But we're seeing the same thing with just about all of them and even our own scene files.

    The guys at MAXON support have said that our machines have enough power. But that we basically need a better networking set-up. Currently everything is going through an 8 port gigabit switch from tp-link. I've even tried a massive 16 port gigabit switch from Amazon which had greater bandwidth and we're seeing exactly the same slowdown. I've turned off all firewalls. Even switched out and tried new sets of cabling (currently using cat6).

    I've seen 10gb Ethernet cards and switches on-line, but it's obviously expensive stuff. Would it even help? I've read about possibly binding two switches together, but don't really have a clue whether that would help either. Or where to start.

    Is there anyone here with larger team render setups that could cast some light on a better networking solution? How are you guys doing it and getting around the bottlenecks? Would something like Smedge circumvent team render and distribute things better? Or is this just how team render works, or does it basically just hate me?

    I appreciate there are render farms and cloud solutions too. But the project we're trying to get this rolling for doesn't have the budget allocated. So would really prefer to leverage the kit we already have, if possible.

    Pretty much close to tears on it anyway and our deadline is looming. Really hope someone can help, I would be incredibly grateful if anyone can.

    Cheers!
     

    grapes_test.jpg

    Share this post


    Link to post
    Share on other sites

    It looks like you're distributing the rendering of a single frame. There's quite a bit of overhead in that process, and the speed benefit will vary greatly by what's going on in the project and how much preparation the scene requires (lots of preparation = less speed benefit because all machines have to perform the preparation).

    What caches are you distributing via the Team Render tab of the Render Settings? In general you'll get better performance by distributing the default caches (all but radiosity, which are huge), but you might try adjusting these options to see what kind of difference they make. I've created a Cineversity Tutorial series explaining Team Render that might be helpful.

    I don't think bandwidth is your limiting factor here - though I haven't tried to reproduce and checked myself my hunch is that the preparation steps that must be performed on each client are hampering your expected performance gains. 

    Ultimately single frame distribution (especially with biased renders) is not as efficient as multi-frame distribution (where the preparation can be done once for a block of frames). Command line rendering and apps like Deadline or Smedge will only support multi-frame distribution, so your only alternative to Team Render is a third-party render engine.

     

    Share this post


    Link to post
    Share on other sites

    To check your network (as long as it's cable based, not WLAN) you could just do a simple file transfer from your render server to all of the clients (not at the same time, though). If this is ok and is =>70MB/s (gigabit can do about 100 or a little more) your networks isn't the problem. 

    I'd agree with what RBarrett said. One single frame produces a lot of overhead if you distribute large files (e.g. the first computer to finish the GI is going to send it to all the others or maybe to the server first then to the others). In a network having multiple actions at once really slows it down. Like with USB sticks, if you copy one file it's fast, if you copy 3 at the same time it's not 1/3 for each file but ... don't know... feels like 1/50th :P

     For single images I usually only use our best PC (mine ^^). Only do still images with a network if the file is pretty large or really takes a long time so that asset distribution isn't the largest part of this.

    For animations network rendering is fast, but ALSO takes some time for the first frames, often it's something like > 10mins, be patient until all the machines have rendered their first picture, after that you'll see the speed you get. If you look at the speed before this happens you're gonna be disappointed.

    Share this post


    Link to post
    Share on other sites
    Guest BTmonocle
  • Topic Author
  • Thanks for the quick reply dudes. 

    Checked cache settings on most projects and all appear to be correct. Everything distributed apart from radiosity. I'll try a range of frames to see if that does the trick when I'm back in the office Monday. 

    One thing I will say regarding network speeds. This is the last switch we tried (http://www.amazon.co.uk/TP-LINK-TL-SF1016DS-Port-Desktop-Switch/dp/B004BU56BS?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_detailpage_o01_s00). Using the grapes scene as an example again, when I checked the console on the client side, the speed of the first machine (Xeon) was only 6.44 MiB\s and 4.33 MiB\s when the second one joined in.  Seems to be miles under the numbers Stefan quoted.

    Turns out a friend of ours here is a networking guy for a local company. So he's borrowing a Cisco box to try on Monday night. No idea what that is, or if it will make any difference. But willing to give it a punt.

    Regarding third party render engines. I know Vray has it's own gui on the client side from our time with Max. But don't other systems like Arnold also use the TR system to communicate too?

    It is an animation job after all. I'm not too fussed for single frame stuff. Hopefully trying a range of frames will do the job.

    Appreciate the responses :)

    Share this post


    Link to post
    Share on other sites

    Don't rely on the numbers C4D tells you. Just find out if your network works with a normal file transfer in explorer/finder. If this works your network should be ok. Cinema often displays low speeds but works. Just look at the frames' renderspeed after the first frame is done on every machine. That's the important part. Compare that speed to your single machine speed.

    Of course there's still a chance that something is blocking/slowing down your C4D transfers, like some antivirus or even a switch.

    Share this post


    Link to post
    Share on other sites
    Guest BTmonocle
  • Topic Author
  • Thanks Stefan, we managed to get it working as expected in the end, turned out it was networking issues after all. Appreciate all the responses!

     

    Share this post


    Link to post
    Share on other sites

    • Recently Browsing   0 members

      No registered users viewing this page.

    ×
    ×
    • Create New...