In this presentation, I will tell the story of how we built a telehealth app in the browser for doctors to see patients virtually during the pandemic, and how we implemented a virtual backgrounds feature so that doctors can always have a professional-looking video background, even when seeing patients from home. I will explain how machine learning in the browser works with TensorFlow.js, and how to use segmentation models to identify a person in an image with BodyPix. I will also cover how to take ML segmentation data and use it to paint a virtual background behind a person on a video call using HTML Canvas compositing techniques - all in JavaScript. Finally, I will demo this feature with a sample video chat web application running on Twilio.

Mark is a Software Engineer for Doximity, working remotely out of Honolulu, HI where he is helping to bring telehealth to the largest professional medical network. Mark is passionate about front-end development and pushing the limits of what's possible in the browser with Machine Learning, WebRTC, WebAssembly, and other emerging technologies. Mark is involved in the Digital Nomad community and enjoys extensive work-travel. Besides work, Mark enjoys hiking, photography, and learning to cook Mexican dishes. Ask him how to hit a mean cut-shot in air hockey!