The buzz at Big Data London hinted at changes to come in 2019 – from new data-driven experiences to the rise of DataOps and the evolution of big data beyond cloud and open source.

If you work with data in the UK, chances are you were at November’s Big Data London event. This was the third year the show has been held, and it was by far the biggest and most wide-ranging yet.

I was there to find out what kind of challenges companies are facing in getting big data and analytics programmes embedded in the organisation, and to get a feel for how things might evolve next year.

The conversations I had, and the presentations I saw, suggest three key developments to come in 2019.

#1 Data will drive completely new customer experiences

Lots of organisations are still getting started with big data and analytics, and the business value of their fledgling initiatives may not yet be proven. Big Data London had plenty of content to inspire them to press on with building platforms that will give their business a true competitive advantage.

One of the new tracks at the event focused on using data to create unique customer experiences. One presentation stood out for me: New Nudges by Alastair Cole, Chief Innovation Officer at Andrews Aldridge.

Alastair showed how customer data coupled with machine learning can lead to the creation of “big ideas, crafted for individuals” – enabling truly personalised products (as opposed to recommendations of existing products) to be created and offered to consumers on the fly.

Few organisations are in a position to be able to do this successfully today, but many are heading in that direction. Our own work with companies like Rank and the Restaurant Group is focused on building platforms that can ingest vast amounts of customer data in real time, to allow machine learning algorithms to be applied to up-to-date data.

That kind of platform takes time, effort and expertise to set up and maintain, but Alastair Cole’s presentation provided a glimpse of the kind of unique value it can deliver.

#2 Advanced organisations will move to a “DataOps” model

Another new track for 2018 at Big Data London was “DataOps”. Just as DevOps has made software development more agile, data ops promises to do the same in 2019 for data-driven activities.

Anyone involved in analytics today will recognise the frustration of trying to meet business demand for instant intelligence, when the processes for gathering, pooling, cleaning and interrogating data come from a previous era and are tortuously slow.

Organisations that are serious about always-on insight, and about applying AI and machine learning to data in real-time, need to completely rethink the way data is handled and delivered in the organisation. DataOps, with its emphasis on responsive, agile processes, seems to hold the answer.

It’s certainly something we’re seeing our customers ask for. We presented a case study at Big Data London about our work with the Rank Group to speed up the process of pooling data, applying and updating machine learning algorithms, and delivering the insights back to the business.

Previously, it used to take around seven months for Rank to make new data feeds available to gain business insights. By building and managing an analytics platform in the cloud, and re-organising IT operations around the delivery of the resulting insights, we’ve been able to bring that process down to hours and minutes.

As AI and machine learning initiatives increasingly emerge from their R&D ivory tower and start to be embedded in the business, the reorganisation of operations to be more data-centric feels like something we’ll see a lot more of next year.

#3 Big data will evolve beyond cloud and open source

For a long time, big data has felt synonymous with the open source movement. If your organisation has a strategic commitment to Microsoft, you might have felt that events like Big Data London were not for you, and that you were missing opportunities to harness the full value of unstructured data.

But that’s changing, and fast. As a Microsoft Gold Partner, Telefónica Tech was privileged to present to the Big Data London audience some new capabilities coming to SQL Server next year.

With purely relational databases increasingly preventing companies from unlocking the full value of their data, Microsoft has made radical architecture changes to SQL Server 2019 to address the challenge. Its new big data clusters capability will enable teams to move quicker, work with a wider array of data, handle massive datasets and augment their code with open-source libraries and projects.

That’s important for another reason, too: it will make it easier to run big data projects on-premises. With many organisations preferring to keep their data inhouse, for regulatory, policy or (perhaps surprisingly) cost control reasons, the ability to pool structured and unstructured data for real-time analysis will be game-changing.

The challenge for data scientists: staying focused on delivering business impact

If Big Data London is any indication – and as one of the UK’s biggest big data conferences, it should be – then 2019 promises to be an exciting one for data-driven organisations.

The challenge for data scientists, though, will be to stay focused on delivering business impact, and not get bogged down in the nuts and bolts of operating and evolving the analytics infrastructure.

Database technologies, analytics and visualisation tools and cloud platforms all evolve fast, and keeping up with the underlying tech may not be the best use of time for a skilled data science team.

10 questions to ask now – and a half-day to focus on the way forward

If you’re currently mulling the best way to operationalise and evolve a big data, analytics or AI program, it makes sense to consider these ten questions before making a decision.

You may also welcome an opportunity to get your data science team together to think about the best way forward. Telefónica Tech would be pleased to organise a free half-day workshop to explore your objectives and current model, in order to uncover the best solution. For more information about what that would entail, email me at dan.perrin@telefonicatech.uk.