https://www.youtube.com/watch?v=sR7hBJGpwQo&list=PLWKf9beHi3TgstcIn8K6dI_85_ppAxzB8&index=57&t=0s ================================================================================ Apply domain adaptation to medical image ================================================================================ * Heart image via MRI * Use domain shift to convert MRI image to CT image * Use network which is trained by MRI data * See whether segmentations can be possible on CT image * Lower domain adaptation results in bad segmentation (1st image from right image group) * Good domain adaptation results in good segmentation ================================================================================ * Domain shift lowers performance * Train network over MNIST dataset * Good classification * Apply model trained over MNIST data to SVHN image data * Worsen performance ================================================================================ Domain shift * Train network over domain A data * Apply trained network over domain B data ================================================================================ * Why performance degradation happens in domain shift? * Due to different data distributions * You can see classifier trained over MNIST doesn't work for SVHN data ================================================================================ * Domain adaptation ================================================================================ Feature adaptation (paper: adversarial discriminative domain adaptation, 2017) * Domain adaptation in feature level * train network * train network by using "domain label" * use discriminator which create "good fake feature for SVHN and MNIST" * Classifier which is trained over SVHN data at first pretrain step can classify MNIST data ================================================================================ Domain adaptation method 2: Combine 2 adaptive strategies * Train CycleGAN first for image level domain adaptation * Use ADDA for feature level domain adaptation ================================================================================ * Images of heart * Different moodalities ================================================================================ * Problem * Domain experts create label for each region of heart of CT image * Domain experts should create labels for each region of heart of MRI image ================================================================================ * Problem 2 * Performance degradation in domain adaptation ================================================================================ * Model arcitecture ================================================================================ * Result (a): without adaptation (b): image adaptation using CycleGAN (less performance) (c): feature adaptation (c): Image+feature adaptation ================================================================================ Workflow * Perform image adaptation by using pixel-to-pixel image transformation (GAN model) * Train generator $$$G_t$$$ which makes CT-looking image * Train discriminator which distinguishes fake-created-CT-image * Use GAN loss to train both networks * GAN loss $$$\mathcal{L}_{adv}^{t}(G_t,D_t) = \mathbb{E}_{x^t \sim X^t} [\log{D_t(x^t)}] + \mathbb{E}_{x^s \sim X^s} [\log{(1-D_t(G_t(x^s)))}]$$$ * Role: create CT-looking image from MRI image ================================================================================ * Pass CT-looking image into encoder to create feature images * Pass feature images into Decoder U to create MRI-looking image * If D_s can't distinguish CT-looking and MRI-looking images, it means good training on encoder and decoder ================================================================================ Add segmentation network: Classifier C, $$$D_p$$$ ================================================================================ ================================================================================