FaceTime, like most other macOS system apps, does not support virtual cameras. See this page for further information.

I have seen that page but it does not appear to explain this situation. That page talks about an inability to load DAL plugins, but in my case here the DAL plugin is clearly being loaded, as evidenced by the appearance of the OBS test card. What's failing is that the DAL plugin is unable to talk to its Mach client counterpart.

Another third party DAL plugin, https://github.com/seanchas116/SimpleDALPlugin, works fine in FaceTime and Photo Booth. So my guess is that whatever restrictions FaceTime is putting on its code, such as maybe using the hardened runtime, prevent the OBS Mac VirtualCam plugin from doing its internal Mach communication.

I tried re-signing OBS.app adding com.apple.security.temporary-exception.mach-lookup.global-name to the entitlements as an array with one element, com.obsproject.obs-mac-virtualcam.server (that's MACH_SERVICE_NAME). Assuming I re-signed correctly, that didn't make a difference. I also tried adding it to a copy of FaceTime.app, which already has com.apple.security.temporary-exception.mach-lookup.global-name set to a list of the 33 Apple Mach ports that it is allowed to talk to. I tried adding com.obsproject.obs-mac-virtualcam.server and re-signing, but then that copy of FaceTime would not launch ("You do not have permission to open the application").

I wonder if there is an alternative method of communication that the OBS Mac VirtualCam plugin could use that does not involve Mach ports. The fact that the entitlement name includes the string "temporary-exception" suggests that Apple would like apps to phase out the use of this method in favor of something else. But I am not familiar with Mach communication or alternatives.

The proper way to do this would be to use an NSXPCConnection which also requires a helper process in the OBS bundle. @johnboiles originally refrained from using this as it would make running OBS/virtualcam without a bundle impossible.

As we need helper app bundles for CEF anyway and an app of OBS' complexity should (IMO) never be run as a lone binary to begin with, I wouldn't mind requiring an app bundle for more advanced features such as browser sources or virtual webcams.

This change however would be something some macOS developer needs to be willing to introduce/maintain, but I see it as inevitable.

With regards to this issue, it sounds like to reproduce this one would have to disable SIP on their machine and that's not something I wish to do (as it should not be disabled in the first place) and that's a use case I wouldn't consider "officially supported".

@ryandesign Would you be able to reproduce this with other apps using virtual webcams that do not require SIP to be disabled (and thus are not considered "unsupported" to begin with)?

I haven't disabled SIP on my machines.

$ csrutil status
System Integrity Protection status: enabled.

Interesting, how did you make it pick up the virtual camera? Because by default it shouldn't.

Oops, my mistake. On this machine, a 2012 MacBook Pro, I had indeed run a script which disabled SIP in order to install macOS 11. Even when booting this machine to a different volume with macOS 10.14 where SIP is enabled, I can select the OBS virtual cam video input in FaceTime, however csrutil status still reports that some protections are disabled which is probably why it works.

On a different machine, a 2016 MacBook Pro running macOS 10.13, the OBS virtual cam does not appear in FaceTime's Video menu. However a different third-party DAL plugin, EpocCam, is in the menu, can be selected, and it does work completely, showing video from my iPhone's camera in the FaceTime window on my Mac when the EpocCam app is opened on the iPhone.

@ryandesign I see you noted it works for discord did you have todo anything special? I found virtual camera on MacOS Big Sur works in many app's like firefox etc but discord just says "No Video Devices" on my Mac Mini

Per my previous comment, it is apparently the disabling of System Integrity Protection that allows it to work and bypass the restrictions imposed by newer OS versions. Big Sur isn't supported on this Mac so I used someone's script to install it anyway, and that script disabled SIP for me, which I had neglected to remember initially.

On a different machine, a 2016 MacBook Pro running macOS 10.13, the OBS virtual cam does not appear in FaceTime's Video menu.

This is probably not a useful datapoint after all because 10.13 doesn't include the hardened runtime, which is seemingly what causes our problem here in the first place.

As far as I've now been able to determine, there is simply no way to do it with apps that use the hardened runtime's library validation feature. Apple says:

We have reviewed your request and have concluded that there is no supported way to achieve the desired functionality given the currently shipping system configurations. If you would like for Apple to consider adding support for such features in the future, please submit an enhancement request via Feedback Assistant

I have also been looking around and found this thread. I didn't manage to make it works, however, I found out that cloning facetime.app to my desktop enabled an extra Camera selection list in Video menu.
I can see OBS Virtual Camera but selecting it does not do anything. The video stream is still coming from `Facetime HD Camera (Built-in).
I'm running big sur 11.2.3 with OBS 26.1.2, and SIP is enabled.