Protect thread setting call

[kpedro88]

The function at::set_num_interop_threads() in PyTorch can only be called once. Subsequent calls will result in an exception.

(The function at::set_num_threads() can be called multiple times, but after the first call, subsequent calls will issue a warning message and will not have an effect.)

This can cause the server to crash if two models are loaded that both specify values for the corresponding configuration parameters:

terminate called after throwing an instance of 'c10::Error'
  what():  Error: cannot set number of interop threads after parallel work has started or set_num_interop_threads called
Exception raised from set_num_interop_threads at /opt/pytorch/pytorch/aten/src/ATen/ParallelThreadPoolNative.cpp:54 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f1f656896c8 in /opt/tritonserver/backends/pytorch/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x59 (0x7f1f65625341 in /opt/tritonserver/backends/pytorch/libc10.so)

In this PR, std::call_once() is used to prevent this (along with a try/catch block just in case).

Testing the above case (two models that both specify values) with this PR shows that the exception no longer occurs and inference can proceed.

The documentation is updated accordingly.

[kpedro88]

Updated to take #158 (review) into account

[whoisj]

This makes sense to me.

Requested @tanmayv25 to provide additional review as he's more familiar with the PyTorch backend than I am.

[kpedro88]

@whoisj thanks for starting the review. I have applied the code and markdown formatting in the latest commits.

[whoisj]

@kpedro88 please read the Triton Contributors Contribution License Agreement. We'll need this completed prior to accepting any changes from you unless you're acting on behalf of your employer and your employer has a CCLA on file with us.

Thank you.

[kpedro88]

@whoisj I have been a signatory of the Triton CLA since 2020, through my employer, Fermi National Accelerator Laboratory. You can see my past PRs to Triton repos.

[whoisj]

@whoisj I have been a signatory of the Triton CLA since 2020, through my employer, Fermi National Accelerator Laboratory. You can see my past PRs to Triton repos.

My apologies, I missed FNAL in your bio. My mistake, and I do see FNAL in our contributors list. Thank you for the correction.