Blogs
Because sharing is caring
Building the Data Plane while flying it
In the data domain you typically have to balance between building the right thing and building the thing right.
The days of being able to spend 6 months or a year on “Sprint Zero” creating your data platform have gone.
One team I worked with called it “building the airplane as you fly it”
Here are 5 patterns I have seen data teams adopt to help them do this.
2024 the year of the Intelligent Data Platform
AI was the buzzword for 2023 and it will continue to be the buzzword for 2024.
I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.
Ask AI
Assisted AI
Automated AI
Adopting these patterns moves a data platform from being a manual data platform, towards a data platform that can do some of the data work for you.
An Intelligent Data Platform.
The 3 patterns of AgileData AI
Having AI embedded in your product have become table stakes it seems.
I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.
Ask AI
Assisted AI
Automated AI
Demystifying CDP’s vs. Data Warehouse’s
In this article we describe the concepts of Customer Data Platforms (CDP) versus Data Warehouses.
The magic of DocOps
TD:LR Patterns like DocOps provide massive value by increasing collaboration across team members and automating manual tasks. But it still requires a high level of technical skills to work in a DocOps way. For the AgileData App and Platform, we want to delvier those...
Iterations create milestone dates, milestone dates force trade off decisions to be made
Data teams struggle to not “boil the ocean” when doing data work.
Use milestones as a pattern to help the data team to focus on what really needs to be built and manage the trade-off decisions for what doesn’t.
Your data team are mercenaries, define your ways of working based on this
Modern data teams are transient, often staying less than 5 years, unlike past decades of long-term loyalty.
Companies should adapt by defining robust Ways of Working (WoW) that endure beyond individual tenures.
Balancing in-house teams with reliable data vendors for continuity and efficiency may also be a useful pattern as part of your WoW.
I’m getting pedantic about semantics
TD:LR Having a shared language is important to help a data team create their shared ways of working. When we talk about self-service, we should always highlight which self-service pattern we are talking about.I'm getting pedantic about semantics. In the Data Domain we...
The Art of Data: Visualisation vs Storytelling
Data visualization is like painting with data, using charts and graphs to make trends and patterns easy to understand. It’s great for presenting data objectively.
Data storytelling weaves a narrative around data, adding context, engaging emotions, and inspiring action. It’s perfect for persuading stakeholders.
AgileData App
Explore AgileData features, updates, and tips
Network
Learn about consulting practises and good patterns for data focused consultancies
DataOps
Learn from our DataOps expertise, covering essential concepts, patterns, and tools
Data and Analytics
Unlock the power of data and analytics with expert guidance
Google Cloud
Imparting knowledge on Google Cloud's capabilities and its role in data-driven workflows
Journey
Explore real-life stories of our challenges, and lessons learned
Product Management
Enrich your product management skills with practical patterns
What Is
Describing data and analytics concepts, terms, and technologies to enable better understanding
Resources
Valuable resources to support your growth in the agile, and data and analytics domains
AgileData Podcast
Discussing combining agile, product and data patterns.
No Nonsense Agile Podcast
Discussing agile and product ways of working.
App Videos
Explore videos to better understand the AgileData App's features and capabilities.
Unveiling the Definition of Data Warehouses: Looking into Bill Inmon’s Magicians Top Hat
In a nutshell, a data warehouse, as defined by Bill Inmon, is a subject-oriented, integrated, time-variant, and non-volatile collection of data that supports decision-making processes. It helps data magicians, like business and data analysts, make better-informed decisions, save time, enhance collaboration, and improve business intelligence. To choose the right data warehouse technology, consider your data needs, budget, compatibility with existing tools, scalability, and real-world user experiences.
Martech – The Technologies Behind the Marketing Analytics Stack: A Guide for Data Magicians
Explore the MarTech stack based on two different patterns: marketing application and data platform. The marketing application pattern focuses on tools for content management, email marketing, CRM, social media, and more, while the data platform pattern emphasises data collection, integration, storage, analytics, and advanced technologies. By understanding both perspectives, you can build a comprehensive martech stack that efficiently integrates marketing efforts and harnesses the power of data to drive better results.
Anatomy of a Data Product
A graphical overview of the components required for a Data Product
Unveiling the Magic of Data Clean Rooms: Your Data Privacy Magicians
Data clean rooms are secure environments that enable organisations to process, analyse, and share sensitive data while maintaining privacy and security. They use data anonymization, access control, data usage policies, security measures, and auditing to ensure compliance with privacy regulations, making them indispensable for industries like healthcare, finance, and marketing.
Free Google Analytics 4 (GA4) online courses
TD:LR There is some great free course content to help you upskill in Google Analytics 4 (GA4) Here are the ones we recomend.Discover the Next Generation of Google Analytics Find out how the latest generation of Google...
5E’s
As Data Consultants your customers are buying and outcome based on one of these patterns – effort, expertise, experience or efficiency.
We outline what each of these are, how they are different to each other and how to charge for delivering them.
Agile-tecture Information Factory
Defining a Data Architecture is a key pattern when working in the data domain.
Its always tempting to boil the ocean when defining yours, don’t!
And once you have defined your data architecture, find a way to articulate and share it with simplicity.
Here is how we articulate the AgileData Data Agile-tecture.
Data Architecture as a Service (DAaaS)
TD:LR Data Architecture as a Service (DAaaS), is it Buzzwashing or not? As is often the case, it depends on your point of view. Our point of view? Nope its a real thing.
Myth: using the cloud for your data warehouse is expensive
TD:LR Cloud Data Platforms promise you the magic of storing your data and unlimited elastic compute for cents. Is it too good to be true? Yes AND No. You can run a cloud platform for a low low cost, but its will take...
Observability, Tick
TD:LR Data observability is not something new, its a set of features every data platform should have to get the data jobs done. Observability is crucial as you scale Observability is very on trend right now. It feels...
DataOps: The Magic Wand for Data Magicians
DataOps is a magical approach to data management, combining Agile, DevOps, and Lean Manufacturing principles. It fosters collaboration, agility, automation, continuous integration and delivery, and quality control. This empowers data magicians like you to work more efficiently, adapt to changing business requirements, and deliver high-quality, data-driven insights with confidence.
The language of data is not so natural
TD:LR The dream is we can just point the machine at our data, ask our question and get a useful answer. With ChatGPT we are closer than we have ever been, but we are not there yet,When Nigel and I first started...
Build Data Products Without A Data Team Using AgileData
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about our AgileData SaaS product and our focus on enabling analysts to do the data work without having to rely on a team of...
How To Bring Agile Practices To Your Data Projects
TD:LR Late in 2022 I was lucky enough to talk to Tobias Macey on the Data Engineering podcast about combining agile patterns and practises with those from the data domain. Listen to the episode or read the transcript....
App Engine and Socket.IO
We wanted to be able to dynamically notify Data Magicians when a task had completed, without them having to refresh their browser screen constantly. Implementing websockets allowed us to achieve this.
I can write a bit of code faster
TD:LR To get data tasks done involves a lot more than just bashing out a few lines of code to get the data into a format that you can give it to your stakeholder/customer. Unless of course it really is a one off and...
The Focus Podcast – Agile Data Governance Patterns
Early in 2022 Shane Gibson was lucky enough to talk to the Focus podcast crew about agile governance in the data domain. Watch or listen to the episode.
ELT without persisted watermarks ? not a problem
We no longer need to manually track the state of a table, when it was created, when it was updated, which data pipeline last touched it …. all these data points are available by doing a simple call to the logging and bigquery api. Under the covers the google cloud platform is already tracking everything we need … every insert, update, delete, create, load, drop, alter is being captured
Three Agile Testing Methods – TDD, ATDD and BDD
In the word of agile, there are three common testing techniques that can be used to improve our testing practices and to assist with enabling automated testing.
Using a manifest concept to run data pipelines
TD:LR … you don’t always need to use DAGs to orchestrate Previously we talked about how we use an ephemeral Serverless architecture based on Google Cloud Functions and Google PubSub Messaging to run our customer data...
“Serverless” Data Processing
TD:LR When we dreamed up AgileData and started white-boarding ideas around architecture, one of the patterns we were adamant that we would leverage, would be Serverless. This posts explains why we were adamant and what...




























