AI in maritime search and rescue is at the core of Zelim’s innovation. In this article, originally published in The Motorship, Co-Founder Doug Lothian shares how the team developed autonomous systems to detect people in the water and build the future of maritime safety.
Doug shares how Zelim is pioneering the use of AI in maritime search and rescue — from detecting a person in the water in real time to building the foundations of fully autonomous safety systems. We’re republishing it here to make it more accessible to our network.
In 2020, when we began developing Guardian – an unmanned craft designed to rescue and recover people from the water – one of the first challenges we had to solve, was how do we find people in the water when there are no humans onboard to do the searching. We were looking for a system that could automatically detect people in the water to enable fully remote or even autonomous search with our Guardian vessel. After looking at what was available on the market, nothing fit the bill. All the existing maritime detection technology was focussed on collision avoidance for the safety of navigation, nobody was focussing on search. At this point we realised that we needed to build something ourselves. A system that didn’t just detect objects for collision avoidance, it would have to be able to detect and classify the object to be useful to a searcher. Therefore, the system would have to know the difference between a human head in the water, a rock, a dolphin or even a lobster pot. At this point we made our giant leap into artificial intelligence.
Science fiction has provided many examples of intelligent machines that can perceive the world around them, detecting and recognizing objects and their relevance to the environment. The reality of machines capable of perceiving and understanding the world began in 1958 when Frank Rosenblatt, with the U.S. Office of Naval Research, unveiled a remarkable invention—a 5-ton computer called ‘Perceptron.’ ‘Perceptron’ hinted at the possibility of automated learning. By the 1980s, advances in statistical methods and neural networks laid the groundwork for modern machine learning, although computing power was a limiting factor that slowed progress.
When we started our journey into AI it was reaching a tipping point. Major tech firms were open-sourcing frameworks that could detect and classify objects on land with increasing accuracy. Meanwhile, GPUs (graphical processing units) were becoming more powerful and affordable, offering processing that could crunch huge volumes of image data in milliseconds. These two developments made it possible for us to train real-time vision systems outside the realm of Big Tech.
However, the sea is a tough environment for machine learning technology. Everything is in motion, sea spray and light reflections can play havoc with vision systems making the task of differentiating a floating human from a drifting bag of rubbish or piece of driftwood huge. Detection systems that worked on land just could not stand up to the challenges of the sea. It required a different logic.
Spotting the Human
Detecting something in the water is one thing but classifying that something as a person is a very different thing altogether. And doing it in real time, from a moving vessel, in unpredictable sea states, adds unprecedented levels of complexity.
A person in the water does not have a fixed profile. They might be wearing dark clothes, submerged, face down, bobbing, rolling, or tangled in equipment. From an AI perspective, that is a moving target in every sense.
So, we began building our own maritime-specific database. Using drones, we flew hundreds of test missions and recorded people in the water from multiple heights, angles, and distances, under varying weather conditions. Every image was manually annotated. We drew bounding boxes around people, labelled them, and fed the results into our training models.
Today, our dataset includes more than 6.5 million labelled objects. Anecdotally, according to the US Coast Guard, it is the largest and most rigorously tested visual dataset in maritime search and rescue.
That validation matters, because it is the foundation for how our models “learn” what a person – an adult, adolescent, child or infant – looks like in the water, not just from one point of view, but from every conceivable angle. You are not just teaching AI what a person looks like, but enabling it to recognise a person it has never seen before, from a distance it has never experienced, in conditions it has not yet encountered.
In the early days, AI models would look at a still image and make a judgement about what was in it. But that approach is flawed, as AI can misidentify wave crests, foam, or a seabird as a person, leading to false positives which, in a rescue scenario, can costs lives.
Instead of making a decision based on a single frame, the technology we have developed looks at how features persist and move across multiple frames. We also added range estimation, using camera intrinsics we can accurately calculate how far the object is from the camera. From that, we estimate its size, and from its behaviour, we can also predict its future drift path. All of this has culminated in a system that does not just detect – it reasons. It watches. It tracks.
Autonomous Ecosystems
As our detection capability matures, its possibilities multiply. If a machine can reliably detect and classify people, it can also detect obstacles, navigational hazards, and security threats. That is the basis for full maritime autonomy, not just for rescue vessels but for all commercial ships.
We have started designing this into new product modules. As an option with the ZOE Intelligent Detection System, we can include ‘Watchkeeper’ capability to deliver continuous visual monitoring for bridge navigators. Watchkeeper acts as an AI-powered lookout, alerting crews to any navigational obstacles to reduce the risk of collisions and accidents.
Shield is another new product module that builds on that foundation to provide greater situational awareness and security, helping crews detect suspicious craft, unauthorised boarding attempts, or potential collisions.
Both modules rely on the same underlying AI engine we began building for man overboard detection. But now they are being used in wider contexts, fusing Radar, AIS, GPS, and Electro Optical and Infra Red visual data into one cohesive operating picture, integratable with modern Bridge Systems (ECDIS).
The Future
At present, we are processing real-time detection at HD resolutions . Processing every frame at 30 frames per second, applying object detection, classification, tracking, and probability estimation within milliseconds, is still a monumental computing task. We are working with GPU arrays and edge processing to push this further.
But resolution is only part of the picture. We are now deploying multi-sensor fusion techniques and expanding our classification range.
It is worth noting that in many operational scenarios today, human decision making still outperforms automation in nuanced judgement, but not in consistency or speed. The US Coast Guard’s own studies show that the probability of a helicopter crew spotting a person in the water – even when flying directly over them – is just 18%. In comparable studies, ZOE has demonstrated 96% Probability of Detection and Recognition, outperforming the human by over 5x. AI systems do not get tired, do not blink, and do not overlook what is right in front of them.
That is why we believe this technology is not just the future of search and rescue – it is the future of maritime safety in an increasingly autonomous maritime world.
