Find you get too much of a performance hit using other recording tools like OBS Studio? Maybe this new tool will help. It doesn't have a fancy name, just called "gpu screen recorder" along with GTK front-end you can use to help manage it.
They claim it's the "fastest screen recording tool for Linux". Why make it? Well, in their own words on how it's different to OBS when paired with NVENC:
OBS only uses the gpu for video encoding, but the window image that is encoded is sent from the GPU to the CPU and then back to the GPU. These operations are very slow and causes all of the fps drops when using OBS. OBS only uses the GPU efficiently on Windows 10 and Nvidia. This gpu-screen-recorder keeps the window image on the GPU and sends it directly to the video encoding unit on the GPU by using CUDA. This means that CPU usage remains at around 0% when using this screen recorder.
In their own testing, they said they saw FPS drops from 30 to 7 with OBS Studio at 4K but with their tool they saw a solid 30FPS. This of course highly depends on your CPU but the point here, is that it uses the full power of your GPU, and seeing a performance loss with OBS on Linux is pretty common.
They have a lot of plans for it including support for AMD and Intel GPUs, the ability to dynamically change bitrate / resolution to match desired fps, see the cursor in the recording and more.
Sounds like a great project, that could be helpful for those of you who want high-performance recordings or something more like Nvidia ShadowPlay perhaps.