Projects to Platform- from Platform to Productivity
In Nov 2019 I presented at a series of Confluent Streaming Events - Madrid, Amsterdam, London, Frankfurt, Zurich and Paris.
Here's a summary of my talk.
Hi, I’m Lyndon (from London) and I’ve been working at Confluent at a Customer Solutions Director for the past 2 years.
In this role I’ve worked across many Enterprises - helping them on their journey of implementing an Enterprise Event Streaming Platform.
I’m going to talk for the next 30 mins on the transition when event streaming PROJECTS merge (or pivot?) to become a PLATFORM.
We will cover the typical adoption journey for Enterprise event streaming and focus on some of the key success factors for optimising business value, which effectively means - making money, saving money and / or protecting money.
Why Listen?
For organizations embarking on the event-streaming platform journey, I will look to address questions such as:
“How do we evolve from PoVs / PoCs or individual projects to support broader enterprise wide adoption?”
“What talent [skills & experience] do we need to set-up an internal Enterprise service?”
“How have other similar successful companies transformed their Operating-Model for an Enterprise wide platform?”
This isn't a technical talk. For many people in this room, you can think of this as a kind of career counseling session. I will give you some hints and tips for accelerating your career around Event Streaming (assuming that’s what you want to do). Or, you might consider this just a cynical way of trying to sell you more stuff :-) That's up to you.
To be clear - this talk supports Confluent's Mission:
To help companies around the world adopt event streaming so they can harness the power of real-time data.
To some extent we have been successful in this mission: Kafka has been adopted by nearly two thirds of the Fortune 100 as businesses across financial services, retail, automotive and more are making event streaming a core part of their businesses…
BUT I believe there's a dirty little secret behind this statistic. The truth is, for some of these organizations Kafka is still only used in small edge cases. It has not made the leap to a wider Enterprise wide Platform (yet). In other words, event-streaming may not truly have ‘crossed-the-chasm’ - to really hit the mainstream (in 2019). The Platform play for Kafka still largely applies to the Digital Natives / Digital Firsts or truly innovative organizations.
And this is a problem ...Because Confluent's Vision is for:
Kafka and the Confluent Platform to become the central nervous system for every modern company.
So, this talk is designed to not only help in our mission - to help companies adopt an event streaming platform - but to also help in our vision - for the event streaming platform to become the ‘central nervous system’
The term central nervous system is synonymous with the term Platform. So, I will start here…
Why is the Platform important and what do we mean?
There is a famous 2011 article by Marc Andreessen on the thesis that software is eating the world. It turns out this was remarkably prescient. This sentence has become the unifying theme in the wave of tech companies of the last decade. It’s is also an idea that unites many of the current trends: ubiquitous mobile connectivity, IoT, machine learning, microservices, the explosion of SaaS apps, and cloud.
We’ve since become familiar with this saying - Every company is becoming a software company…
Take FS for example; 10,000 of Capital Ones 40,000 employees are software engineers.
JPMC: Employs over 50,000 people in technology and has $10B+ technology spend.
Car companies are investing heavily in Digital,
Retail - the same
...and this is true across the board.
Recently our CEO took this a step further and claimed that every company is becoming Software. I could talk about what this means - and the benefits - but Jay Kreps does a better job of this than me - in his KSSF keynote. I suggest going to watch this and to read his blog.
https://www.confluent.io/blog/every-company-is-becoming-software
And to emphasise what this really means, let me show this slide - you may have seen something like this on social media - it is a few years old now.
Each of these companies has automated a process into Software. They have removed the person…and created a Platform business model.
So, what is a Platform - and what are the benefits?
Definitions vary. I quite like these McKinsey definitions.
https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/five-fifty-platform-plays
This is our simplified definition of a platform.
We have Data Stores (Mainframes, Hadoop, Data Warehouses), Logs, 3rd Party Apps and Customer Apps (based on Microservices architecture.) - The Universal Event Pipeline.
When we have the Contextual Event-Driven Applications - We have the Central Nervous System.
As an example, in Retail you have Point of Sale (POS) transactions. Previously this 'event' - a Sale - might have 'fed' a Finance system, which then integrates to other systems. Now a POS event can be consumed from many systems; Inventory Management, Supply Chain, Logistics, Dynamic Pricing, Customer 360 Profile, Web - for Personalised Targeted Contextualised offers etc.
This enables a company to become software, which in turn enables the Enterprise to make money, save money and / or protect money.
So, how do we get there?
So, let’s start with looking at the typical adoption journey for event streaming.
The first thing to address is that when it comes to this new paradigm, it’s not all or nothing.
Adoption is incremental. It often begins gradually, and then later accelerates as the value of the paradigm becomes universally apparent in your organization.
Here is the most common pattern we see by which our customers adopt Confluent Platform - either via our software solution or managed service.
First, it begins with early interest, where a developer learns about Kafka and perhaps starts with open source Kafka or our free developer edition. Importantly, Confluent free developer edition includes KSQL, which can accelerate your learning curve. At this stage your organization will have the technology available to start working with your data as a continuous stream of events.
The next phase is sending event streams into either Confluent Platform or open source Kafka, connecting systems through those event streams, and possibly building an application. At this stage you will have started building out your event pipeline and making events broadly available to systems. The value here is generally from the aspect of having a universal event pipeline. Here is also where you really want to think about the future, as this is where the value of open source Kafka usually ends - unless you are willing to commit to heavy internal development.
The third phase, and a truly critical one, is the creation of 1-3 event-driven applications. Here, the application leverages stream processing, and creates a business outcome that was unable to be attained in the old world of data as static records or individual messages. This is critical as it marks your organization starting to gain the full potential of event streaming.
After the first application is in production, things often accelerate. This is when organizations enter the flywheel of adoption, where the more you add, the more powerful the platform becomes. In addition there is a positive feedback loop where more apps create more events, which in turn enables more apps, and the cycle goes on. This is the Network effect. We’ve seen over 30 new applications leveraging Confluent Platform in just a matter of weeks.
Finally, as your event pipeline becomes ubiquitous and event-driven applications become the enterprise standard, you will have transformed into a contextual, event-driven business. You will be responsive to your customers and be able to create business outcomes in a way that was never possible before. Here, your Event Streaming platform becomes the Central Nervous System of your Enterprise, both powering your applications and transporting all your enterprise events.
So, moving from 3 - 4 is where things often accelerate. We see a transition from Projects to Platforms. I see many organisations that have reached stage 3 - and struggle to get to a full stage 4. So, this is where I will focus this talk.
So, what are the biggest challenges for organizations moving from Projects to a Platform? There are a number… for example, we have to identify the use cases (business pain-points) - where benefits will be realised… There are the technical challenges - but Interestingly, the hurdle is rarely a technical one.
My argument is, the biggest challenge is… PEOPLE.
From a people perspective the journey looks like this.
And of course, I'm not the only one to find this. In his article in 2011, Andreessen called out talent as a major blocker to the software revolution. Quote: “many people in the U.S. and around the world lack the education and skills required to participate in the great new companies coming out of the software revolution.” Andressen went further by stating “every company I work with is absolutely starved for talent.”
This is a quote from the NHS report after the “biggest IT failure ever seen”, which cost £10bn. In the post mortem analysis, the NHS simply stated: “digitising effectively is not simply about the technology, it is mostly about the people.”
We see this time and again. As an example, This McKinsey & Co. study found, when they surveyed Executives, they consistently point to People, Skills and Management challenges as the top issue for infrastructure modernization. When I see this, I see PEOPLE: Talent Gaps / Change Management / Understanding (education) / Operating Model.
I also want to introduce the 80/20 rule, or The Pareto principle.
As a quick aside, in 1896 the Italian economist - Pareto, noted the 80/20 connection while at the University of Lausanne: Pareto showed that approximately 80% of the land in Italy was owned by 20% of the population. It turns out that many natural phenomena have been shown empirically to exhibit such a distribution:
Wealth - 80% owned by 20% people
Taxes - 20% of people pay 80% of taxes
Language - 20% of vocab makes up 80% of speech
Moon craters - 80% of the area of craters made up of top 20% of craters.
This is partly why teams should be split to the size that can share two Pizza’s - you can manage numbers to avoid this rule. But this is a worthwhile consideration when thinking about People.
So, what to do?
We can see a Platform is an important business model. And whilst Confluent isn’t a Transformation Consultancy - and we don’t profess to be experts in operating models - but from working with leading Enterprises here are 3 important tips.
First up: Set the Vision.
Get the PEOPLE on board.
This requires education but you also need to think about your audience.
On our adoption journey - we often see a bottom-up play… If we superimpose a typical org structure onto this… we see the following:
In stage 1 - the main stakeholders / owners of the Vision - are the ‘Workers’ - developers, data architects etc.
In stage 2 - the main stakeholders become middle-management
When we move to stage 3 - we transition to Senior Management - and the mission critical use cases get senior visibility - but it’s often still seen as just infrastructure.
To bridge to 4 - we need to set the vision at the CEO and C-suite.
We need a top-down as well as bottom-up meeting of Vision - of becoming software - of a platform play.
The key here is messaging and education → Sharing the vision.
Tip 1. Set and communicate the VISION.
Different stakeholders will understand the vision differently. We need to articulate the vision for the right audience.
Confluent can help here. We have an engagement model to support this: HQ visits, Tech Talks, Meetups, Executive Sponsor, white papers, Access to Confluent Evangelists/ Sales Engineers, Streaming Assessment - and events like this.
As an actionable item, we need this team to translate what they're doing with Kafka into business language to help set the vision. Here's an example of a translation map.
Tip 2. Consider the Operating Model.
The second tip is to address the Operating Model for Enterprise event streaming. I’ve written a white paper on this topic. And this deserves a whole talk in it’s own right.
Firstly, ensure you have the Expertise. Let’s consider WHY expertise is so important.
Behind Apache Kafka’s power is its architecture. Distributed, stateful systems have a number of characteristics that make them difficult to operate and maintain. Even though Kafka's architecture is elegant and simple compared to competitors, some of the complexity cannot be avoided without deep understanding of the system. You want to focus on building applications, not infrastructure. You need people who understand the subtle differences in architecting around events, not RDBMses.
Expertise helps address 3 key areas: 1. Complexity - Streaming Systems are distributed. 2. Non-trivial capacity planning. 3. Large surface area to cover.
You also need to architect your Business Operations - around the PLATFORM Business Processes.
You no longer need to be silo’d by functions - Sales, Marketing, Supply Chain, Logistics etc.
You should organise around common Customer journey patterns and data journeys. This is a big change.
With an agile workforce you can add Squads to satisfy additional ‘Products’ or business use cases, we see the introduction of Chapters, as shown in the diagram below.
Chapters are groups of employees with similar expertise, or skill sets, that can be spread over several different Squads. Examples include a team of Java developers or distributed system data engineers. These teams would have a Chapter leader, who is also a "regular" squad team member.
Guilds are similar to Chapters and provide a forum for the sharing of learning between multiple users of the same technology, or multiple implementers of similar applications. Guilds tend to be more organic and wide-reaching “community of interest” (or special interest groups - SIG’s. Guilds also work across different Tribes which may included non-event streaming aspects of the business, such as integration with the ERP or CRM teams.
Confluent Expertise helps address these challenges. It includes the following; SE Advisory, Professional Services, Training, TAM + Partner Ecosystem…e.g. Marionete. And there’s important events - which are great opportunities to hire talent.
As an actionable item, think about how your team is organised and perhaps implement special interest groups, or chapters / guilds.
Tip 3. Technology - a Complete Event Streaming Platform.
And 3rd tip = Technology. This might sound odd coming from a Software Company - we’re placing our platform 3rd. But remember - we are talking the transition from stage 3-4 - and I sincerely believe to get to a Platform Play, the Vision and Operating Model need to be in place BEFORE we talk Technology and the Confluent Platform.
Obviously - as a platform - you need to think about a Complete Event Streaming Platform - in terms of Development and Streaming Processing + Operations and Security - and it’s obviously beneficial to buy vs. build… We are also seeing some more mature organisations offer an Event Streaming as a Service internally to their business units.
And our Fully Managed Cloud service - a lot of customers can effectively leap-frog to stage 3. The fully managed service often takes away the people - set-up and operations challenges that take up most of the focus on stages 1 and 2.
And finally, to conclude…
To compete in the Digital world (where the company is software) - to help make money, save money and / or protect money - an Enterprise needs a Platform Model.
The main challenge in getting to this point is 'People'.
Our tips are to
1. Set the Enterprise Vision.
2. Align the Operating Model and then
3. Think about the Technology.
I hope this talk is useful and please get in touch if you'd like to discuss further.