Analyzing retail data to tackle the industry's most enduring challenges

Foreword

From the adoption of e-commerce to the growing popularity of discount retailers, the retail sector is facing more disruption now than it has at any other time in its history. The consequences of these trends cannot be ignored. For example, in the UK, retailer margins have fallen by 40% in the past 10 years, while in the US, 8,600 retail stores are expected to close in 2017 alone.

In recent decades, the data available to retailers, and their ability to collect it, has grown exponentially. From ePoS transactions through customer footfall to forecasts and inventory, the data is readily available. However, the ability to analyze and draw value from it has progressed at a far slower rate. With the rise of new technologies, such as smart sensors and the Internet of Things, the volume of data that retailers collect will only continue to increase whilst they struggle to make sense of the information they already have.

In this whitepaper, we look at the ways that you can gather and analyze your retail data to overcome the current and future challenges of retail sales growth, productivity, and profitability. We explore the positive effects of an insight-driven approach on your operating margins and look at how better collaboration within your business and across the supply chain can significantly increase your sales.

We will also share our insights on the role of data in improving on-shelf product availability and look at how you can save millions of pounds of cost from your waste and markdown process through better data analytics. Discover how you can better use data to quickly align your business goals and drive year-on-year sales increases within months of engagement.

Harnessing your data more effectively to drive in-store efficiency

Bricks-and-mortar grocery retailers are being squeezed from all sides – growing e-commerce, declining customer loyalty and the rise of discount retail are all impacting traditional retailers’ profits. There have been seismic shifts in major retail markets; in the UK Aldi now has a market share of 7%, overtaking Waitrose and the Co-Op.¹ Bricks-and-mortar vendors need to up their game if they are to compete in this new retail landscape.

Extracting value from data

Data can have a big role to play, if used correctly. The problem is that many retailers are data rich and insight poor. By using data to generate excessive reporting, they drown their retail operations in it. It is not unheard of for a store manager to receive over 100 reports a day and be expected to decipher them. With no insight to help them, their ability to execute actions and compete is impeded.

Industries like manufacturing and oil and gas have actively embraced deeper analytics, artificial intelligence (AI), and the internet of things (IoT) to drive operational effectiveness, while retailers continue to lag behind. As Paul Boyle, CEO at Retail Insight, asks, “retail scan data opened the door to what the world now defines as business intelligence, but has it really pushed on as far and as fast to areas such as predictive analytics?”

According to John Paul McNeil, COO at Retail Insight, “Grocery retail has traditionally focused on the creation of heavy dashboards sat on top of large data warehouses at a time when headcount is reducing every quarter. There is not the resource to dig into hundreds of reports; from top floor to shop floor our clients are telling us that they simply want to know which actions to take to deliver highest value in the quickest time and with as little resource as possible.”

Better insights, better business decisions

However, as the data streams grow, adding more information could risk increasing the seemingly endless amount of backwards-looking reporting retailers must attempt every day. Collecting more data does not mean you are getting more insights, it merely means you are getting more data. The rear-view mirror approach to business intelligence and reporting may tell you what has happened, but you then want to know how you can do something different to make a measurable change to your performance. If this can be automated, then it is a productivity game-changer for retail.

We should be focused on extracting insight through intelligent automation – building machine-driven insight that can guide, or even take, action rather than simply providing data visualization on what has happened to a workforce which does not have time to look at it, much less form it into something approaching value.

John Paul McNeil
COO of Retail Insight

Global management consultancy firm McKinsey & Company believes those retailers that employ an insight-driven approach to sales and in-store efficiency can develop “a deeper understanding of consumer and shopper behavior and embed these insights into the way they manage every product category.” Retailers in the grocery, drug, and DIY sectors that have used the insight-driven sales approach have achieved a sales uplift of 3-5% and a net margin improvement of 1-4% in 6-18 months.⁴

Companies that are unwilling or unable to improve how they analyze data will fall behind their competitors that do, and by a considerable margin. McKinsey Global Insight estimates that retailers who exploit data analytics at scale across their organizations could increase their operating margins by more than 60%.⁵

There is a large prize for automating this insight and the pressure in the market means retailers have no time to waste. Retailers looking to drive sales and productivity quickly should look to employ a trusted partner specializing in operationalizing retail insight. This will ensure that you get the most from your data in the shortest time possible, allowing you to concentrate on executing an optimized retail plan.

If we can provide a positive answer when retailers ask whether this information is enabling them to do a better job in less time, or even to eliminate tasks having no value to the business, then we are on the right track to significant productivity gains.

Paul Boyle
CEO of Retail Insight

Using data for better collaboration

A recent Work.com study found that 97% of employees and executives agreed that the level of collaboration directly impacts the outcome of a task or project. When a team or department is collaborating smoothly it creates greater flexibility, speed, and engagement.⁶ But collaboration is more than an open-plan office – underlying siloed behaviors can undermine the effort. Collaboration requires a commitment to adopting a shared set of goals and agreeing to implement the changes needed to achieve them.

Data and predictive technologies have an important role to play in helping to facilitate the changes required to create value-driving collaboration.

“In recent years, we have seen brick and mortar retailer margin performance flatline to decline whilst sales are depressed. Data from the British Retail Consortium shows significant reductions in headcount already and predicts further cuts by up to one third in the next decade.⁷ This sort of change often leads to fear, distrust and a ‘keep-your-head-down’ mentality appearing across organizations as individuals look to protect their jobs. In times like this, innovation naturally falls by the wayside,” explains Paul Boyle, CEO of Retail Insight.

“Within the current retail organizational model, innovation is easier to come by when it is delivered from one department and with one clear owner,” Boyle adds. “If it stretches across two departments then there is a greater chance we will run into competing agendas or have disinterest as the initiative benefits one more than the other.”

Work together for the greater good

According to Accenture, successful collaborators can generate up to 10% in operating margin improvements, raise store-shelf stock rates by 5-8%, reduce inventories by 10%, cut logistics by 3-4% and yield a 5-15% reduction in manufacturing costs.

With the potential for results like these, it can be difficult to see why retailers are not putting more effort into collaboration. Dr David Waters, CPO at Retail Insight, explains, “Of course, it can be due to internal politics, but it can also be caused by data quality and, at times, the data not finding its complement to make it massively more valuable. Although there is lots of data being stored at head office, it does not always get refined to reflect the reality of what is happening on the ground.”

One example of this is found on in-store promotions. Waters continues, “How often do retailers use their data to really assess promotional effectiveness from top floor to shop floor? We often look at designing promotions in merchandising without connecting our history of in-store execution. We might have had conflicting measures of success – for example, a merchandiser focused on hitting a sales and margin target whilst a store manager worries about waste and inventory.

“In that instance, the merchandiser may have bought big and run a successful event, but left the store with a significant stock overhang that the store now needs to hold or markdown. A simple connection here would bring the data to life for everyone and help them understand the potential drivers of success and failure.”

Do not be surprised if you encounter resistance to collaboration that has its origins in concerns such as:

• If I share what I am doing, good or bad, then could that be used to point the finger at me? 
• If I collaborate and we win then how do I recognize my contribution? Do we split the benefit?
• If I collaborate then how do I keep control? 
• Ongoing, who will be the ultimate owner of this connected system? Am I making myself redundant?

Investing in better retail data, and data connection, is a key way to align your internal and external targets, making sure everyone is on the same page. According to a report by Retail Week, the most successful retailers share information and interpret it both internally within their organization and externally with their different partners: suppliers, third-party logistics providers, carriers, and more.⁸

For William Dankworth, ex-VP of Grocery at Kroger Co and now retail industry advisor, time, system limitations, lack of automation, and culture can be impediments to data sharing and sustainable collaboration.⁹ Knowing how to overcome these time and resource restraints is key for retailers trying to achieve collaboration.

“That is where Retail Insight can come in,” explains Waters. “We spend every day looking for ways to connect data to glean better insight and help our clients create value. In a recent project for a major retail client, we connected all head office and store merchandising data and attributes in one holistic model to understand the actions that delivered sales growth and those which did not. We identified that we could improve sales by over 6% and that their cross-departmental pain sat in event execution – all departments came together, galvanized by the data and insight, to solve a meaningful problem for the business.”

What role does AI play in retail innovation?

Artificial Intelligence (AI) and Machine Learning are redefining online retail. But many are yet to fully harness this technology in physical stores, where, particularly in the grocery category, the majority of their goods are still sold.

Translating the success of online AI and machine learning to brick-and-mortar stores may be the key to unlocking retail innovation, but increasingly narrow margins make justifying such expensive and unproven technology a risky gamble.

The opportunity

E-commerce providers have embraced the reality that AI and the Internet of Things (IoT) will become the consumer’s personal assistant as they move through the path to purchase. AI-powered devices, such as Amazon’s virtual assistant, Alexa, and Clickatell’s AI chatbot, Touch, both remove barriers to create seamless customer experiences.

At the same time, consumer behavior is also changing. Postmillennial consumers expect a higher level of convenience and immediacy in-store, where they are increasingly shopping with singular intent and using smartphones to check prices, compare brands, plan meals, and so on.

“Grocers cannot afford to stand still,” says Paul Boyle, CEO of Retail Insight. “The only thing we can be certain of is that the retail rate of change, or more specifically the rate of diversification, is not going to slow any time soon. Innovation, flexibility and agility are now the watchwords.”

The future is coming

Hyperlocalization, powered by AI, offers the potential to tap into this changed consumer behavior and engage with individuals in precise, contextual ways. In-store, grocers can turn to beacons, digital signage, price tickers, and on-shelf displays while tracking movement, brand interactions, promotion exposure, and purchases.

If this real-time data could be used in conjunction with AI, then retailers might be able to work out who their customers are, when they come to the store, and what they want. All this can be achieved while ready to deliver a more personalized and customized service for customers on arrival. Where traditionally retailers have been engaged in business intelligence informing them of what happened yesterday, they are now moving to prediction – to forecast what will happen tomorrow.

So far, we have centered attention on how we drive improvements in the customer proposition, but what about the operating model? For an industry rooted in manual processes, introducing AI to improve grocery retailer operations could be transformative.

Dr David Waters, CPO at Retail Insight, says, “We are now starting to connect the dots on data to deliver actionable predictive insight at the operating level. Our hope now is that the functions within retail can collaborate to take the value – what will happen in a particular store tomorrow or this weekend and what resource will be needed to manage demand, and supply, in store? How should each store be merchandised to meet consumer needs and can this model be operated effectively?”

UK grocery spending grew 3.7% in the first quarter of 2017

What is holding grocers back?

UK grocery spending grew 3.7% in the first quarter of 2017, primarily driven by inflation, according to research from Kantar Worldpanel.¹⁰ All 10 major grocers are in growth, but the Big Four are still losing share to discounters – and overall retail spending is in decline, as customers continue to feel the pressure on income.

In this ultra-competitive marketplace, do grocers have the time and resources to invest in expensive technology that is still largely unproven in the market?

The criticism that has always been leveled at big data projects is that they typically take too long to return value on the significant cost invested.

McKinsey recently claimed that UK retailer margins have dropped by 40% across the past decade and this has driven an increasingly short-term outlook on projects and investments.¹¹ Retail Insight CEO Paul Boyle explains, “We need to think differently for our clients. Although innovation efforts can never be underwritten for success, retailers are looking for partners to help them lower the risk profile of investments by working to define a journey that delivers value in stages and to be truly agile in their development whilst working towards the grander vision of, say, AI-powered prediction.”

“Most mathematicians we talk to wonder why everyone is throwing so much money into a concept that has been around for a long time, but maybe has not had the compute power to demonstrate value. There is still a ‘so what’ factor,” reveals Waters.

Although AI and prediction should be possible within retail – the sample should be large enough – its success remains dependent on its underlying data and algorithms, which are determined by past events and historical learnings.

“A machine learning engine does not say ‘I think I have got something missing in my code’, all it can do is optimize from what it already has; it cannot invent the data. We must remember that there is still a huge gap between what we want AI to do and what it can do, but as computing power grows the possibilities are growing too,” states Waters.

The success of AI is intrinsically tied to acquiring more, and also more accurate, data. For example, a tool used to predict product availability may indicate when it believes items need to be replaced. “But you need real evidence to confirm whether items were actually on the shelf or not, in order to improve the algorithm,” says Waters, “and that is why we spend so much time creating the effective feedback loops in our prediction engines, making them more accurate every day.”

Do grocers have the patience for technology that is still largely untested? And are their resources not stretched by the vast amounts of data they already have to process?

It starts with a single step

Many analysts believe it is only a matter of time before more complex AI processes become sophisticated enough to be used widely in-store. However, in today’s cost-constrained retail environment, it is vital that value is delivered through the journey, and not just the destination.

Boyle concludes, “Right now at Retail Insight, we are focused on using AI and machine learning to help remove millions of dollars of cost from our retail clients’ repeatable processes and to improve productivity. This might not be a direct translation from e-commerce, but people are now more awake to the possibilities.”

“However, we are always clear that it is about applying the right mathematical model for the right problem at the right time. Machine learning and AI are very exciting areas to explore, but they are only part of the modern mathematician’s armory. There are so many ways to take value from the data and as retailers lean into new technology, they should partner with someone who can help them gain from both the journey and the destination.”

Transforming sales and profit through sales-based measurement of on-shelf availability

If we believe the most recent – and comprehensive – study of global out-of-stocks, then retailers are currently running at an average rate of +8%. This is a figure that has remained largely unchanged across the past 20 years.¹² In simple terms, this equates to store shelves being empty for 28 days per year or the equivalent of every store closing its doors for an entire month.

These figures are typically defined by gaps, but not all availability gaps are created equal - something many measurement tools do not consider. If milk, carrots, and chicken fillets are missing, then that seems like only three items. However, they represent a potentially significant availability problem. The items in question are high demand, high velocity items, and their unavailability is likely to disappoint customers.

Retailers often believe that if the shopper is faced with an out-of-stock scenario they will likely buy a substitute item, but that is not always the case. In their global study of out-of-stocks, Corsten & Gruen assess that at least 36% of cases result in shoppers abandoning their purchase or defecting to another store.¹³ It is not just a lost sale though, as this scenario could lead to a shopper deciding never to come back. Here, it is the lost lifetime value of that customer at stake.

As global grocery retail feels the squeeze from multiple disrupting factors, sales and margin growth is increasingly hard to come by, but there is a real opportunity to deliver value to the business and its shareholders. Retailers, suppliers, and customers all recognize the problem – if the product is not on the shelf, then the shopper has no chance to buy it.

Why has there been so little progress?

Progress can be hindered by the sheer complexity of the issue at hand. Hundreds of stores stocking many thousands of products provide millions of opportunities for out-of-stocks to occur. It is often easier for suppliers and retailers to focus on agreeing another volume-driving promotion, than to engage in delivering a sustainable improvement in sales through an improvement in availability.

But the solution is relatively simple in concept. First, measure your availability performance to provide visibility, transparency, and drive accountability for the results. Then, use this visibility to identify the key issues and make the major interventions required to ensure enough of the products that shoppers want to buy are stocked on the shelves where they expect to find them.

The retailers employing this approach more effectively than their competitors are already winning – and the prize is worth playing for.According to research from IHL Group, the cost to the global retail sector caused by problems with on-shelf availability is approximately $634 billion per year.¹⁴ And, across the past three years, Walmart was reported to have set its sights on addressing a potential $3 billion out-of-stock problem¹⁵.The momentum the retailer is currently enjoying suggests real progress is being made on addressing the fundamental challenge of availability within its store operations.

However, most existing measurement systems are ineffective when it comes to identifying the issues and can even be counter-productive in driving the behaviors required to make progress.

The cost to the global retail sector caused by problems with on-shelf availability: $634 billion per year

Using your sales data to help measure availability

Creating a sales-based measure of availability goes a long way to helping. This method looks at a massive set of sales data for every item, in every store, for every hour to understand what the store should expect to sell. For example, if a store expects to sell 20 avocados an hour and suddenly there are no avocado sales through the checkout for a full hour during the day, it is safe to assume that something is obstructing that sale. Namely, it is currently not on the shelf.

John Paul McNeil, COO of Retail Insight, explains, “A sales-based measure of on-shelf availability will identify if you have not sold as much as you think you should have. We think that is fairly clear because we can see the history behind the analysis, and we are tracking it at such a fine level. We can look at the sales data, look at what should have been expected from those items and stores, then provide retailers with a quantified lost sales opportunity.”

With the introduction of advanced data analytics, our views on the measurement of availability have dramatically evolved into a management-by-exception-based approach and facilitated a transition from more people instore, to optimizing the productivity of the current workforce. Using sales data, and your enterprise data, to assess availability can provide alerts and direct people to where there might be a problem. Then, it can be fixed quickly..

Paul Boyle, CEO of Retail Insight, concludes, “The proof is there. We have put data at the heart of the availability challenge within our clients and created a sales-based measure and tool, driving real, aligned action and value from top floor to shop floor. In one of our most recent implementations of this methodology we have seen our system deliver a directly attributed year-on-year sales benefit of 1.8%.”

“And when we launch, we do not leave – our subject matter experts will continue to partner with our client to deliver even more value every day.”

Optimizing markdown in retail to reduce waste and increase sales

When discussing the future of retail, and in particular grocery retail, there is a tendency to focus exclusively on the radical shifts required to compete in this disrupted market.

Brick and mortar shops are taking their sales into the digital space, while companies develop and invest in omni-channel ways to communicate with customers. Dealing with these challenges is expensive, and requires large investments in the areas of cost, capability, and cultural change. But there are some quicker wins to be had from focusing on the last 50 yards of retail.

For most grocery retailers the potential to extract value exists everywhere and can often be realized by taking iterative steps from basic to fully-optimized or preventative solutions. In the UK, the government-funded Waste and Resources Action Programme (WRAP) estimates that in the UK over two million tonnes of fresh produce are lost or wasted each year in the supply chain alone.¹⁶ Action to reduce or prevent this could save retailers upwards of £400m every year. Across the US, this figure will run into the billions of tonnes.

Dr David Waters, CPO at Retail Insight, says, “Obviously, there is a lot you can do to minimize the impact of out-of-stocks and waste through a more carefully designed and joined-up plan on range, space, pricing, forecasting, distribution, and labor. But that is a big plan with the need for significant coordination and change management. It could stretch into years of effort. Meanwhile, there is value to go after right now.”

Striking a balance

Getting the balance right across the business between availability and waste, and between long- and short-term goals, is no mean feat and that is before attempting to build the technologies required to deliver a total preventative solution. However, there is still a lot that can be done beforehand.

Data analysis can drive availability measurement, insight, and action to close gaps tactically and systemically. At the same time, it can be used to understand sales rates, stockholding levels, and the impact of price movement to appropriately mark down product price to clear before it ends up in the bin.

As Waters explains, “When retailers mark product down, if at all, usually it is in simple increments: first by 25% then 50% and to clear at 75% off. Introducing the ability to apply an item-store specific, optimized price reduction can be done quickly, adapting the price reduction dependent on factors such as stock on hand, time of day, typical store sell-through…etc. This can happen without the need for heavy resource investment on the retailer side and the returns have proven to be huge in every market launched.”

In the case of the continued retail challenge of balancing fresh product availability and waste, the ultimate aim would be to always have just enough availability with little-to-no retail waste. But this is a complex stakeholder challenge to consider with many competing objectives:

• On entering most grocery retail stores; the shopper is faced with walls of fresh produce. Their presentation, and their availability, typically sets the scene for the shopper’s experience of the store. If they cannot find what they want with the right date, will they turn and leave?
• When considering the retail operation, can this availability be provided at all costs? When everything is supplied in excess to guarantee availability then do we expect the waste bill to rise too, when our research shows that it can already account for up to 4% of revenue?

• From a corporate stakeholder perspective, and from the view of society and the environment in general, are we considering our corporate social responsibility objectives? Are we making and disposing of too much?

Start small, think big

Paul Boyle, CEO at Retail Insight, explains, “Sometimes the desire to deliver the perfect solution to the problem acts as a paralyzing factor. Perfect is the enemy of good.”

“Start with an appropriate data-based target,” he continues, “then develop a robust measure and put a basic, simple plan in place to capture quick value. If we only ever wanted an optimized solution from day one, then I think we would be waiting a long time for that day to come.”

To conclude the point, Dr David Waters adds, “Again, this is one of those examples where you apply the right mathematics at the right level and get the desired result in very short order. Sure, we can go big, we can throw AI at the forecasting problem long-term to achieve a near-perfect balance of availability and waste, and we believe we have the foundational models to achieve that long-term.”

“But right now there is long-term value in prevention and short-term value in reduction, so in this time of disruption and margin pressure let us take as much benefit today so that we can fight on tomorrow.”

Projecting forwards

The retail industry is going through a period of intense change, and established models are being acutely disrupted by a shift in social, economic, political, and technological trends.

The technological trends are having the biggest impact on the sector. Developments in technology are meant to bring about productivity changes to facilitate a more efficient workforce and increase margins. However, the shift to a multi-channel model has only brought more complexity and cost as retailers try to balance an asset-heavy brick-and-mortar business with the need for a lighter, more flexible online model.The initial response by retailers has been to deleverage. They are simplifying their asset base while reducing headcount in a bid to fund investment in physical stores and grow online models in the short-term. However, competition from pure-play e-commerce (which is both asset and resource light) means traditional retailers must put data at the heart of their operation if they are to survive in this new environment.

There are serious productivity gains to be had by properly utilizing data and insight. McKinsey and Company argue that “companies that are more data-driven are 5% more productive and 6% more profitable” than companies that are not.¹⁷

Surely that is a prize worth striving for?

To begin properly harnessing and using that data, companies need to know how. First, all useful data must be made available, and this can only really be achieved by breaking down the barriers that exist across and between organizations, driving meaningful collaboration to achieve a clear and common business goal.

In recent years, the CMO and CIO have been given the charge to turn data into growth, but that assumes that the heads of operations, supply chain, human resources, and so on have nothing to add to the plan. As David Waters, CPO at Retail Insight, explains, “There are countless examples of the best-designed head office plans falling down at store execution. However, when it comes to reviewing why the plan did not work, the store-level operational data is not considered. If we do not collaborate on data and insight, then we are doomed to repeat the failures of the past.”

Collecting the massive amounts of data that a retailer holds is vital, but it is only the first step – you then need to be able to deliver insight. Modern retail was one of the first industries to espouse the use of data and many believe it gave birth to business intelligence. Despite its benefits, this approach focuses more on the volume of data and reporting, providing lots of visualization for what happened yesterday without giving managers the insight they need to deliver results. As one store manager of a leading European retailer said, “I am sent over 450 reports each week and I am expected to have a detailed knowledge of each of them.”

John Paul McNeil, COO at Retail Insight, explains, “Retailers want to empower their staff with smarter tools and better insights to allow them to work in a more efficient and productive manner. The focus has to be on bringing the data together and then being agile enough to not only analyze it quickly, but to use the insight to make decisions faster and take Chapter 2 action that drives better business results.”

Once we have mastered the art of using data to drive action and correct the immediate, and enduring, issues of availability and waste, we can then look to use data to predict future trends. Many organizations, including retail, have started using predictive analytics programs with the view that it is as simple as collecting the data and then setting to work on finding hidden patterns, even when it is unknown what business problem those patterns might help solve.

Waters argues, “The failures seen in big data initiatives have typically come when they start with too much hope and with little definition on the bounds of the problem. At Retail Insight, we believe that you start with the business problem and then work out what data, technologies and analytical solutions are required to solve it.”

Boyle adds, “We appreciate that the potential is massive, and we are building a longer-term grand vision, but we fully believe that there is always near-term value to be delivered as well. Indeed, in our most recent implementations of our availability management and fresh waste reduction solutions, we were able to deliver many tens of millions of dollars in savings within months of launch whilst working towards the client’s long-term vision.”

Despite industry research suggesting that retailers recognize the value in insight and consider it critical to their performance, many feel that they lack the capability to effectively take value from the data they are collecting. As retail headcount decreases whilst complexity increases, it is important that retailers partner with the right specialist retail analytics partner to unlock the insight to drive meaningful value from the huge data collected every day - enabling you to identify the relevant patterns in your data to deliver the best business results.

Steps to success:

• Make all data available through better collaboration across business units
• Structure your internal and external resources to use insights more effectively and allow you to make better, quicker business decisions
• Invest in better data analytics to improve the insights gained from this data
• Select your partner with a retail data analytics specialism, enabling you to identify the best path to improved performance through data and domain expertise
• Focus initial analytics plans against solving defined issues and delivering demonstrable value quickly – once you have a method that works then plan for the bigger, longer-term issues