Introduction

In January 2025, I began working on a deep learning project to segment and analyze biological neurons. This project not only encompassed my master’s thesis, but two additional course projects, part of a collaborator’s PhD, a conference presentation, and neuroscientific publications. The subject of this collaboration between one of UCPH’s AI labs and neuroscience labs was to use modern machine learning approaches to analyze a massive dataset of neural microscopy data. I cannot possibly describe the project in its entirety here in an organized way, so I will focus on the largest part: using computer vision techniques to segment biological neurons.

The data I worked with was Two-Photon Calcium Imaging (2PCI) recordings. This technique uses fluorescent calcium indicators to track calcium ions inside cells, these act as a real-time visual proxy for the neuron’s electrical activity. I was initially presented with a collection of 2PCI recordings from human neural organoids— that is, extracted stem cells which have been reprogrammed into neurons. A couple of months into the project, I introduced a second corpus of 2PCI recordings of in vivo mouse tissue.

On this page, I am giving a high-level overview of the project, intended for anyone curious to be able to read and understand. If you would like a more detailed or technical description, feel free to email me and I’ll send you a copy of my thesis or one of the supplementary reports

Background: U-nets and Organoids

Neural Organoids are human stem cells reprogrammed to become living 3D neuronal tissue in a lab. They give us a non-invasive window into the human brain. To record them, we use 2PCI, which results in video files of the cells flashing as they activate.

To extract usable data from these videos, researchers traditionally have to draw manual boundaries, called Regions of Interest (ROIs), around every single cell body (soma) and connective branch (process). This manual labor is a massive bottleneck which limits scalable 2PCI studies. While some open-source automated tools exist, they are built for standard, dense animal brains. When applied to sparse, lab-grown organoid data, they fail completely. Other generic segmentation models, either for segmenting supposedly ‘anything’ or for cellular specific models also failed when applied to this data.

In order to train a model that could accurately segment these organoids, I turned to the U-Net. A U-Net is a type of Convolutional Neural Network (CNN) that has become the gold standard for biomedical image segmentation. It gets its name from the U-shaped architecture: a contracting encoder shrinks the image down to learn broad, abstract features, while an expanding decoder restores the image to its original size to output a pixel-perfect mask. Crucially, it uses “skip connections” to pass spatial details directly across the network, allowing it to remember exactly where structures are located.

Training U-Nets

The challenge with training this data was that I was only given 12 pairs of ground truth annotations to train my models on. So, my first models were trained on just 9 images and evaluated on a set-aside 3. I used an open-source framework for training U-Nets called Yucca, which is developed at the Pioneer Centre for AI where I was doing my thesis. Without any modifications, the baseline U-Net trained on just 9 images performed quite poorly, plagued by severe artifacts and a low F1/DICE score. This is completely expected for a first round of training with sparse data.

The first solution I implemented was data preprocessing. Instead of treating the 2PCI recordings as standard full-color images (which is what the neuroscientists used to annotate the data) or reducing them to grayscale, I split the red and green color channels into distinct “modalities.” This single change drastically improved the model’s baseline understanding of the cells as it separated two separate calcium indicating channels into their own ‘modalities.’

The second, and more impactful solution, was heavy data augmentation. This involved applying various transformations to both the ground truth images and annotations to turn a single training pair into many. Such annotations included rotations, crops, intensity modifications, etc.

By applying these three transformations sequentially, I synthetically expanded my tiny 9-image dataset into over 10,080 unique training samples! This combination drastically improved the segmentation performance, making the results much less distinguishable from the human annotators.

Later on, my organoid dataset was thankfully increased by the annotators, allowing me to train models on 19 images and evaluate on new test sets. Again, this influx of ground-truth data improved the performance quite a bit. Having a new test set also assured me that the models weren’t overfitting to a small target.

Also, running all of these tests resulted in heaps of models and evaluation data. I highly recommend using a tool like Weights & Biases to keep track of all the run data, system metrics, and artifact versioning.

Training U-Nets: Transferability

A few months into the project, the scope expanded. I was handed a second dataset, this time consisting of 2PCI recordings taken from the brains of living mice (in vivo).

This new dataset was similarly sparse. I only had 10 manually annotated ground-truth images to work with. Just like with the organoids, training a baseline U-Net on just 7 images produced noisy, artifact-heavy masks. However, applying the exact same three-step augmentation pipeline (rotations, crops, and intensity scaling) dramatically improved the mouse model’s performance.

By this point, my organoid dataset had been increased to 22 images. We know that trying to apply in vivo models to organoids produces poor results, but often training model on both domains can improve performance in the other.

Mouse neurons and human organoids look completely different under a microscope. Mouse tissue is highly organized, dense, and full of obscuring blood vessels, while organoids are sparse and scattered. I trained three sets of models: one strictly on organoids, one strictly on mice, and a combined model trained on all 26 available base images.

I evaluated all of them using their F1 segmentation scores. Training a model on both mice and organoids simultaneously actually did result in a better model overall. The combined model outperformed the domain-specific models, proving that the U-Net was learning fundamental biological structures (like somas and processes) rather than just memorizing the specific background noise or domain-specific artifacts of one dataset. The improvement wasn’t as great as that of the augmentations but still worthwhile.

Training U-Nets: Synthetic Data

This last step composed a significant portion of my thesis. The core challenge was always obtaining high-quality training data. Augmentations were great, but what if we could synthetically generate an infinite batch of 2PCI recordings to train on?

To do this, I needed base geometry. I pulled over tens of thousands of highly detailed 2D neural masks from the MICrONS project (a massive database constructed from electron microscopy data). One portion of the MICrONS project is to create a complete 3D model of a mouse visual barrel cortex. To get the masks I simply used their API to access the 3D model and take random 2D slices of the object. These masks would serve as ground truths which the pseudo-2PCI images could be created from.

I initially experimented with Generative Adversarial Networks (GANs), specifically a semantic image synthesis model called SPADE, to generate the images. While it worked reasonably well, GANs inherently require large amounts of training data to learn how to generate new data, which somewhat defeats the purpose of trying to escape the data drought. Moreover, models trained on GAN-generated data consistently underperformed at the downstream segmentation task.

So, I turned to a physics-informed approach. If we know the physics of how a 2PCI microscope captures light, we can computationally simulate its artifacts and noise on our perfectly clean MICrONS masks. To simulate this, I built a pipeline that digitally mimics the physical quirks of a microscope:

• Point Spread Function (PSF): Using the microscope’s known numerical aperture and laser wavelength, I calculated the exact Gaussian blur required to simulate how photons diffract before hitting the detector.
• 3D Depth: I applied a radial gradient to mimic light curvature and artificially dimmed the center of the neurons. I also added heavily blurred background noise to simulate deeper, out-of-focus neural tissue.
• Activity Variance: Because not all neurons fire at the same time or of the same intensity, I randomly assigned base intensities and variances to each individual cell structure so they wouldn’t look uniformly bright.
• Photon Shot Noise: Because the Photomultiplier Tubes (PMTs) literally count photons, they introduce specific statistical noise. I used a Poisson distribution to mimic this effect.

I ran this simulation to generate 500 individual “frames” of noise, and then averaged them together just like the real biological data. The result was a pipeline that could take a flat, perfect mask and output a realistic, completely synthetic 2PCI recording with a pixel-perfect ground truth label attached to it. While synthetic data didn’t ultimately beat out standard augmentations for absolute F1 scores, we believe models trained this way actually began identifying biologically authentic somas and processes that human annotators had completely missed.

This highlights an important problem in computer vision (and ML as a whole), becoming too focused on chasing high DICE/F1 scores. We are always comparing the performance of the model to that of human annotators, but those annotators themselves are imperfect. In the segmentations provided by the U-nets trained on these synthetic images, we noticed that the model would correctly identify neurons in particularly dark regions which the human annotators missed, or particularly small ones which humans couldn’t realistically be expected to annotate.

Results and Conclusion

The final U-Net model, which was put into production and is being used in published neuroscience articles, uses all three of the improvement techniques shown above. An application of the method described here for Schizophrenia research was presented at AIS25, a summit about how AI is being applied into other scientific domains. You can see the accompanying poster above or read the full abstract to get an idea of how the approach is being used to gather meaningful results.

As I mentioned in the intro, the segmentation model is only the foundation of this project. It allows us to extract individual neuron activities, which can be used for myriad investigations. Some of the exciting off-shoots I’ve worked on since include:

• Building algorithms to automatically differentiate structures between somas, processes, and synapses.
• Extracting activity traces for individual neurons for signal analysis to measure bursting, spectral entropy, and SPIKE-synchronization.
• Using Random Forest Models to classify disease status based entirely on those extracted neuronal features.
• Training LSTMs and Transformers on the in vivo mouse data to predict physical behaviors (like whisking and movement) directly from the neural activity traces.

The field of Machine Learning owes a lot to neuroscience, decades ago, biological neurons inspired the first artificial neural networks. Today, we are using artificial neural networks to study biological ones. The practicality of this framework was ultimately demonstrated by its incorporation into an upcoming publication on schizophrenia research, proving that data engineering can push the boundaries of what’s possible in applied medical research.

< Return to home