our journal

• Neil Avery
• 11 February 2020
• Developer Journal

What tools do you use for Kafka DevOps? Take a look and comment if you see anything missing!

• Neil Avery
• 09 January 2020
• Developer Journal

As part of the logscape-ng project (details soon to be set forth), I’ve been setting up the github project to use CircleCI. One of the goals of the project is to adopt best of breed technology and build a completely serverless log analysis platform that runs in your VPC — in the free tier ;) . It is using Quarkus (because my twitter friends cannot stop talking about it), CircleCI and serverless cloud technologies.

• Neil Avery
• 12 December 2019
• Developer Journal

Back in 2009 Damian Guy and I had the mad vision of building a liquid compute, self healing infrastructure upon which you build and deploy apps. We wanted to do it at scale; across thousands of machines. We devised a technology called VScape. The problem with VScape was observability; its a distributed system. Not only did we need to understand what it was doing but what the users app might be doing/behaving (a user app might collect metrics from a Database, Nginx webserver etc). The solution was simple; lets build an app on VScape that did log aggregation; we called it Logscape.

• Neil Avery
• 09 May 2019
• Confluent

So far in this series, we have recognized that by going back to first principles, we have a new foundation to work with. Event-first thinking enables us to build a new atomic unit: the event. Storing events in a stream and connecting streams via stream processors provide a generic, data-centric, distributed application runtime that you can use to build ETL, event streaming applications, applications for recording metrics and anything else that has a real-time data requirement. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. It offers time travel, a source of truth, scaling, consistency and many elements that we can rely on. All of these, and more, lead to design patterns that can be used and reused.

• Neil Avery
• 27 February 2019
• Confluent

With serverless being all the rage, it brings with it a tidal change of innovation. Given that it is at a relatively early stage, developers are still trying to grok the best approach for each cloud vendor and often face the following question: Should I go cloud native with AWS Lambda, GCP functions, etc., or invest in a vendor-agnostic layer like the serverless framework?

• Neil Avery
• 13 February 2019
• Confluent

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Once settled on the event streaming approach, I’ll provide a high-level dataflow of how we design systems for payment processing at scale using this approach.

• Neil Avery
• 31 January 2019
• Confluent

The world is changing. New problems need to be solved. Companies now run global businesses that span the globe and hop between clouds, breaking down silos to create seamless applications that synergize the organization. This continuous state of change means that legacy architectures are insufficient or unsuitable to meet the needs of the modern organization. Applications must be able to run 24×7 with 5-9s (uptime of 99.999%), as well as be superelastic, global and cloud native. Traditional architectures simply cannot meet the challenges of real time and extreme scale.