Introduction
Retailers and brands have for some time been able to operate stable e-commerce platforms that more or less replicate the search, browse and buy experience of Amazon. Anyone not looking to do anything other than replicate the Amazon shopping experience are happy with the vendor e-commerce platforms they have chosen.
There are however a cohort who seek to innovate and differentiate. They seek to connect with customers in different ways and provide more helpful and meaningful guided shopping experiences. These more innovative companies recognise that developing more differentiated experiences that are more useful to customers and store associates will make for a much more rewarding shopper experience and higher customer lifetime value. Retail innovation is however very difficult for the majority on legacy e-commerce architecture.
The headless revolution
There is a buzz around micro-services, APIs and headless commerce in the e-commerce industry today. Many larger and complex retailers have been forced to turn their vendor e-commerce technology into a Frankenstein-like architectures to overcome some pretty significant problems with the legacy e-commerce technology that is now either slowing them down or totally preventing innovation.
When I worked for M&S in the first half of the decade, I got a good glimpse into what the future of the store and the consumer shift to mobile will look like. It was clear to me then why a move to micro-services and API driven e-commerce was a critical step to making sure the digital business was fit for the future.
We were fortunate at M&S in that we were in a business that had convinced shareholders that investing significantly in digital was necessary. There were lots of experiments that we tried. Three quarters of these experiments failed and quite often because of the cost of building and running the technology was too high. Since then the technology has moved on a long way. There has been the advancement of the cloud native, multi-tenant micro-services based vendor e-commerce platforms. There have also been some fabulous advancements in front-end developer frameworks that make building experiences a far more creative process. Current technology now makes the job of building and iterating these new digital experiences a whole lot easier. Some of the things we tried in the first part of the decade are now much more economically viable than they were at the start of the decade. The market for headless commerce is finally gaining momentum.
Building digital experiences at M&S
We spent somewhere in the region of £200m on the M&S digital transformation which was one of the largest ever in the retail world. As well as re-platforming all e-commerce technology from Amazon to an owned and managed technology stack, M&S were also going through a period of digital experimentation of which I’ve not seen repeated by any other retailer since. This was something I was privileged to be a part of – a brief and pleasant interlude in my career in the software business.
At M&S we were building digital customer experiences to help us sell products from core product categories both online and in store. We deployed these on to a variety of device types from consumer devices and computers to in store touch screens, kiosks and store colleague tablets. The buying process a customer would go through was distinctly different for each department in the store. The buying process for a piece of filled bedding such as a pillow is very different from buying a table or a sofa. Likewise purchasing beauty products is very different from picking a suit for a wedding.
We wanted to experiment with consumer mobile apps and within stores we wanted to transform the store into a place where colleagues and customers were leveraging digital to provide a more fulfilling and experiential shopping experiences. This would be a place where store associates would be more productive and customers more likely to leave the store having found, purchased or ordered products. In a world with much more choice we needed to make both store associates and customers to enjoy being in the store and have a reason to come back.
Overall we experimented with around a dozen apps – everything from the core consumer mobile apps to in store virtual shopping rails, virtual makeovers and 3D room designers. The above photo shows a concept store in Kalverstat, Amsterdam. In the background there is the virtual shopping rail – a column of three screens that are all touch enabled allowing the customer to flick through life sized garments. Just in front of that is our 32 inch touch screen order point which was an immersive app that allowed customers or store associates to search, browse and order product. The screens in the store were also powered by a very sophisticated digital signage system that could use contextual data from e-commerce to change the content on the screen according to the context of the environment in which they were placed. Colleagues would be armed with tablets to completed orders.
The headache
The architecture of the e-commerce platform meant we were very much cutting against the grain to deliver and iterate customer experiences. The problems centred around the constraints of the e-commerce platform. The e-commerce platform was the place all apps had to interface with to search & retrieve product information, taxonomy and inventory levels as well as place orders. The generation of e-commerce platforms that M&S had invested in weren’t really designed for anything other than powering the website.
There were some fundamental problems with most of the e-commerce technology of this era: 1) the platforms didn’t have native interfaces (APIs) from which you could easily access the master data and 2) the building of front-end customer experiences required you to build, test & release the entire website application including all the backend logic no matter how small the change – this to ensure you didn’t break something in the process.
In a demanding and ever-changing retail landscape where customer expectations were continually changing and where rapid experimentation was becoming increasingly important, these systems massively constrained us. If it hadn’t been for the huge investment we had at the time, we wouldn’t have got very far. We had to build what the vendor platforms didn’t cater for.
APIs power the digital apps
After completing a digital transformation on to new vendor technology we were operating two e-commerce platforms, one for UK and one for Europe. Both didn’t have native APIs at the time.
There were a dozen or so digital experiences that depended on the APIs being present from in store kiosks to the mobile consumer apps. These all needed a way to get at data that was locked in the e-commerce platform and checkout and purchase product.
Product & search – our first APIs
Because there were no APIs on either we had to resort to extracting data from these platforms and build out our own API based services on which these applications could interface. The way we achieved this was a bit of a kludge. It meant extracting the full product catalogue from the e-commerce platform overnight and loading it into a cloud database on top of which we put a lightweight API that could access the core product information and taxonomy. Search connected directly to Oracle Endeca which powered the site search and filtering. Because these APIs were to be used across a number of applications with separate products teams we put in place an API Management Layer (Apigee) that would secure, control access and manage the performance of the APIs.
Checkout
Checkout was a complicated customer journey for a large department store like M&S. It needed to fulfil orders from a variety of different stock locations. The checkout journey was very complex and building APIs on to two e-commerce platforms to support the checkout journey would in itself have been at least a 12-month programme and several million pounds.
We were forced to do something a little messy, brittle and tactical. We chose to scrape & transcode our website checkout to fit the context of the application. Transcoding was an approach that meant downloading a pages from the website and on the fly changing the HTML to suit the needs of the current application i.e. make it fit the display and look consistent. The transcoding was all done in the cloud by a SaaS platform called Usablenet.
It was enough to get you live with these applications but left you with a high cost of ownership and higher risk of breaking things. Every time you changed the checkout experience on the website, you’d have to change the different checkouts you’d spawned (transcoded) from it.
Website
The website team was frustrated by the lack of agility they had. Every time the website team wanted to make a front-end change, the monolithic Java application that powered the site had to be changed by a back-end Java developer whose core skillset was Java and not writing user interfaces. We had to get our front-end developers to write the front-end pages in HTML and then send them to the back-end developer to integrate into the Java application before the front-end developer could test. This was error prone and time consuming as the front-end developer couldn’t rapidly iterate and test their front-end code without depending on another team. It was therefore also hard to attract and retain talented engineers.
The e-commerce Java application was one ageing, giant, monolithic application full of all the logic powering the website shopping experience. The whole application had to be released together and go through a manual regression test process that tested all parts of the website and dependent front-end applications that hung off it. This release cycle would last 6 weeks even for the simplest front-end changes because you would have 10-15 developers all working on different parts of the one application.
The cost of manual testing was huge. There were between 15 – 20 testers involved in a 6 week release cycle. It was impossible to build automated tests that would quality asssure all functional and non-functional aspects of the application. This in spite of all the modern DevOps innovation out there in building, testing and releasing code in a cheaper, easier and more reliable way. The architecture of the application just didn’t allow for it. The technology was invented 10 years before the DevOps revolution and had been largely neglected by vendors.
IBM’s offering was so far behind what we needed. The other e-commerce platform used by our competitors was Oracle ATG which suffered from the same problems. This on-premise technology was old and the vendor wasn’t innovating fast enough to keep pace with the market demands of omnichannel retail.
Our technology was not fit for purpose and we had to do something about it.
Off with the head
Nobody who had to work with the e-commerce platform was satisfied, it was a legacy technology that had aged badly and was no longer compatible with the requirements of modern e-commerce inside a big retailer. Despite vendor promises, they were not innovating quickly enough in this fast-changing world.
Only six months after rolling out the new UK e-commerce platform on IBM Websphere we took the ambitious decision to cut the head off it. Cutting the head off basically meant separating the front-end code from the back-end so that the front-end experience could be iterated fast without the need to make changes to the backend and incur a long slow and expensive release cycle. We really were having to bastardise the new technology platform and decapitation was the only way. We might also be able to hire and retain better engineering talent who had been put off working with the ageing technology.
I left M&S in late 2014, 6 months after we started this work. Checking back in the with M&S team several years later, they were still sawing their way through Websphere Commerce and also wrestling with the ageing Adobe Content Management System (AEM) which was a traditional CMS suffering from similar problems and causing them huge productivity problems. I’ll also talk about headless content management in a later article.
I was back on the vendor side in late 2014 and was starting to observe the same approach we had followed being used by other tier 1 retailers across Europe and North America. Like us they were also stuck on ageing e-commerce technology stacks.
In the next post I’m going to examine the technologies that I wish had been available to us at M&S at the start of the decade. Retailers will now be looking closely at the early adoption of these technologies that are currently in the process of crossing the chasm from early adoption into the mainstream.
Headless commerce is not for everyone and retailers must think very carefully about their strategy and their digital maturity before making these investments.
