Farmers are unwittingly collecting mountains of information every year, but how can you make use of it all? Oliver Mark explores the curious world of big data.

Harvesting data may seem a trivial matter compared to harvesting grain, rape or beet but as the years go by it will get increasingly important. Already, some farmers are collecting lorry-loads of information from yields, soil-mapping, fertiliser applications and weather data and it’s an area that can only expand.

So what is big data? It’s a relatively new buzzword that encompasses any big mass of information churned out by sensors or other recording equipment. In farming, that information could range from fungicide applications to dairy feed rations and from irrigation to slurry spreading.

When did it start?

The output of farm-related data has snowballed since precision farming practices like variable rate application and yield mapping came on the scene.

Agriculture has always been an industry where numbers are important – whether it was how much nitrogen you applied, how much fuel you used or how many hours since the tractor’s last service – so it lends itself nicely to being a generator of lots and lots of data.

Where does this data come from?

Farm offices are beginning to be bombarded by stats on soils, machinery, animal health, weather and crops.

At the same time more and more machinery is sprouting sensors, cameras and other slightly Big Brother-ish equipment to provide operators with up-to-the-minute information on yields, forward speed and fuel use.

Other sensors, such as those used in Deere’s Field Connect, can be buried in the ground to monitor soil moisture. Meanwhile, grain store temperature sensors are churning out info round the clock and high-end weather stations can tell you the exact temperature, rainfall and humidity on any day in the last five years.

Most new combines are able to record some level of yield data and many have a GPS receiver, so you know exactly where the bad places are. Soil samples are taken more frequently and modern day management means information is stored on computers and tablets rather than scribbled illegibly on to notepads. This has made data more accessible and easier to share than ever before.

More recently, telematics has encouraged extra reliance on data services, particularly for harvesting equipment. It provides a way of measuring every detail of a machine’s performance – where it is, what it’s doing and how much fuel it’s using – and relaying it back to the farm office.

How much data can we collect?

The data we collect is measured in bytes but there’s really no limit to how much we can store. Across the world we produce 2.5 exabytes each day – the equivalent of 33.6m 32GB iPhones.

Where is it stored?

We’ve traditionally stored most of our farm information locally on computer hard drives. However, the world is now moving to storing stuff in the cloud – servers accessed by a network with nothing, visually, to see.

This means information can be accessed from anywhere at any time. It also means you don’t need to be in the farm office to view farm information and can increasingly see it on that most ubiquitous of devices – the smartphone.

Storage space isn’t an issue any more – the cloud network has pretty much limitless space.

How do we make use of it?

All of this information is crunched and processed by algorithms, which do what pencil-wielding mathematicians used to tackle but do it a zillion times quicker. These calculations act like a sieve, ditching the information we don’t want and keeping what we do..

It’s the algorithms that add real monetary value to data collection. They transform pages of machinery coding into clear, interpreted information that can be used to produce things like maps or yield prediction models.

The technology probably lends itself best to arable farming. Growers can monitor fields on a square metre by square metre basis, applying nutrients only where necessary and treating disease problems only in areas that require it.

But livestock farmers will also gain – automated feeding systems are already adjusting the quantity and type of feed served out according to the condition, weight or lactation stage of livestock.

Data services could benefit machinery management, too. Telematics systems send real-time information from working machines back to the farm office, allowing a single person to track a whole fleet of machinery.

Sensors dotted around a combine, for instance, can now spot a bearing that’s running too hot. They can send a warning to the local dealer, who could have a replacement ready even before his old one has gone bang.

Who owns this data?

Data ownership is a woolly area. Often it’s the company that produces the algorithms that claims ownership of the information and that can put people off.

Lots of companies would like access to information about your farm. Grain traders would pay a lot of money for real-time crop yields straight from the combine and chemical companies would love to get an accurate picture of which weeds are prevalent on your farm before bombarding you with adverts.

It really comes down to whether you trust the company that owns the database not to share the information they hold. John Deere, for example, insists it will keep all information private.

Which companies are involved in data processing?

Plenty of companies have started flexing their big data muscles. Claas has quite a few combines around the world using its telematics package and Deere uses its Field Connect service to gather information on weather, soil and crops to determine future irrigation decisions.

Chemical company Monsanto is also a big investor in data. It paid over $1bn for a weather data service called Climate Corp last year, which it hopes will improve farmers’ decision making with field-level and up-to-the-minute forecasts. The company reckons it will be possible to double yields by focusing on crops on an acre-by-acre basis.

Other, smaller data service providers are popping up, too. Companies like Ursula, which makes the Scout agricultural drone, see a big market in processing data.

Is farmer knowledge about to be replaced by algorithms?

The big danger with big data is that you come to over-rely on the information. Farming by satellites and algorithms, rather than experience and knowledge, could potentially take the office-bound farm manager to a new level of efficiency.

But it’s no substitute for traditional qualities like intuition, experience and seat-of-the-pants skill.