Growing up I had the same dentist from childhood to adulthood. My dentist’s office was run by Dentist Chung (in Vietnamese I called him Bác Sĩ Chung – which means Dr Chung translated directly) and his sister running the office.

The office was in Garden Grove, in between the Korean and Vietnamese districts. Walking in I would always smell the incense from an herbal shop next door.

The office looked like it was from the 1970s. They had this really old but comfortable couch and constantly played oldies music from the local radio station.

I distinctly recall being afraid as a kid going in, and somehow the office manager convinced me if I did a good job with a cleaning I could someday get the dentist’s chair. With my warped sense of rationalizing things, it all made sense and I calmed down.

When I was in early high school Dr Chung said, “you should think about getting braces and fixing your underbite.” I really had no issues with my teeth so far, but I entertained his proposal. I went to an Orthodontist consult.

The Orthodontist I saw was in the heart of Little Saigon – the Vietnamese area of Westminster. When coming in I waited in the reception area for a bit, where the Orthodontist admitted me in the office.

He asked me to bite down and said pretty quickly – “class 3 malocclusion jaw surgery – recommend jaw surgery.” He explained to me that the process would be to remove my 4 wisdom teeth, have braces for 2 years, have jaw surgery, and then have braces again for potentially another year. He didn’t explain much any pros and cons and ushered me away to talk to the assistant for more details.

In another room, the assistant put on some DVD of the process of dealing with class 3 malocclusions. It meant that I had an underbite, and what they needed to do is remove my wisdom teeth to make space, and then crack my jaw and move it back. The recovery would involve sewing my lips (?) and going on a liquid diet for a while.

The assistant also said that some people liked having this jaw surgery because of improvements to their facial profile. She also mentioned that some people don’t even recognize them after the surgery.

The assistant ended with saying, “You know, Vietnamese are a superstitious bunch, so some say that doing jaw surgery will change your destiny!”

Okay, count me in for not believing in superstition, but really that is the absolute worst thing you could say to a teenager after getting a quick 5 minute consult, a gory video on the treatment of an underbite, and somebody saying it will change your destiny. At that point, I decided not to go along with my surgery and went along my merry way.

A couple years after the consult, I called the dentist’s office to book an appointment, and I was told the dentist had a heart attack! He evidently had been eating a pretty unhealthy diet (I know correlation isn’t causation, but he did eat McDonalds every day for lunch). Fortunately he bounced back and started working again.

A couple years after the heart attack, he actually had another heart attack and this time fatal. When he passed away, my family went to his funeral and saw his grieving sister, and the dentist’s daughter who I talked on and off with throughout going to the office. Oddly enough, the dentist’s daughter did a quick internship at one of my old startups back in the day.

After grieving the loss of my dentist, there were the practical issues of finding a new dentist. Pausing for a moment, I remembered, my optometrist’s brother (whose parents live next to my parents) was a dentist.

Dr Tan Huynh was also in the heart of Little Saigon, but when I drove into his office, they had  computers that could do x-rays, and an efficient staff to make cleanings and appointments way easier. I had realized at that point I had been going to Dr Chung’s office with technology from the stone ages.

With the first consult, the dentist asked me to bite down and asked if I considered braces and jaw surgery to fix my underbite. This time being older, I peppered him with questions on pros and cons. He mentioned my teeth were functionally fine at the moment, but in the future I might not be able to chew as my teeth wore down. Asking what age I might not be able to eat, he threw out what seemed to be the random number of 60.

Remembering the experience at my last Orthodontist, I wasn’t convinced the pros outweighed the cons (eg – cons meaning my destiny would change).

When I moved up to Vancouver, I was faced yet again on finding a new dentist. Jason recommended me to visit an office nearby, where Dr M was the first to see me.

He did the whole consult and analysis, but this time they took pictures and some fancy 360 xray scan. He brought up again my underbite, and we again talked through the pros and cons. I asked whether I should try to fix it and he said a lot of people have underbites and just manage it. Apparently when eating I push food through my back teeth immediately.

During the pandemic when I got my first cleaning I saw Dr F, a younger dentist who was one of the co-owners of the office. She saw my bite and asked if I wanted to fix my underbite, and after the 4th mention in my life it got my thinking a little bit more seriously about it. This time she said Invisalign might be able to fix it.

I came back to another appointment after my cleaning to get an Invisalign consult. They did some scans and because of the pandemic they wanted to limit in person meetings, so the follow-up was a zoom call.

Dr F proceeded to say that she initially thought she could take out my middle bottom tooth, but to fix my underbite.   However she concluded Invisalign wouldn’t work and that I should see an Orthodontist.

This time I was a little more open to it because I was no longer traveling as a consultant during the pandemic, and wearing a mask would make it pretty easy to hide the fact I had braces.

Weeks later I saw the orthodontist Doctor D and they did the initial analysis. He basically said I have two options. First, remove 2 wisdom teeth, braces for 2 years, jaw surgery, then braces for 2 years. Second, remove 6 teeth, braces for 2 years and you are done.

I peppered him with questions on the pros and cons health wise, and he said functionally both would lead to the same outcome. He said the jaw surgery would change my profile, but would come with more risks since it was a surgery. I decided to go with option 2.  I also wondered why when I was a teenager I wasn’t presented with a non jaw surgery option, but I’m guessing it was because the technology of modeling these outcomes weren’t available.

Dentistry is an interesting field because most dentists and orthodontists can’t tell you definitely what will happen with your teeth in the future. It all seems to be what risk/reward you are comfortable with.
As part of the assessment I had to pay $500.  If I chose to move forward with braces they would credit my account, but if not, I would lose it.  I think sunk cost fallacy nabbed me this time as this pushed me over the edge to do a final commitment of the decision.

Before putting on braces, and I had to get 6 teeth extracted.  To ease the pain, I got 3 extracted from my regular dentist, and 3 extracted from an extraction specialist doctor.  Let’s just say, the extraction specialist finished the entire job in about 30 minutes while my regular dentist took about 1.5 hours.  My regular dentist felt so guilty taking so long she gave me her cell phone number and told me to call her if I had any post extraction complexities.

The process of wearing braces involved seeing the orthodontist about every 6 weeks for an adjustment, and compliance to get the results you want.  In addition to braces, you have a wire running across and little hooks where you can attach rubber bands to.  Throughout the process compliance meant always wearing and rotating the rubber bands as needed as well as avoiding eating really hard food (like nuts), to avoid breaking your bracket.  Slipping up on compliance inevitably leads to a longer total process.

When I saw my Orthodontist, I noticed I was the oldest person in the office as it was mostly kids and teenagers.  Often I would overhear my Orthodontist sternly warn the kids that they weren’t being compliant by either not brushing their teeth well or not wearing their rubber bands. I would then hear parents berating their children in one sentence, and in the next sentence begged them to be compliant.  It usually ended with the parents trying to guilt trip their children by saying seemingly unhelpful things like, “don’t you want good teeth like your brother.”

Getting braces as an adult is a bit different as I was on a mission to be compliant and to finish it as soon as possible because I paid for every penny of it.  Psychologically, something different clicks in your head when it is your money on the line.

The initial side effects I had were teeth sensitivity.  There were times hard food was difficult to eat (like sandwiches, cucumbers, steak, etc), so I bought these tiny tots scissors originally intended for parents to use when cutting food for their babies.  The scissors were an obnoxious bright blue color, but I liked it because it was compact and had a case.

One time I had a business meeting with a customer at a restaurant and when the food came I took out the scissors.  The person next to me paused and asked why I had bright blue scissors.  I explained to him the whole dental situation, and then the whole table caught wind of the conversation and asked me about the scissors.  It was a bit awkward in the beginning, but then the whole table spent the next hour talking their dental issues.  Also through this experience I learned bringing scissors is generally helpful at restaurants if you are sharing food.

2.5 years later (6 months behind schedule mind you), I had an appointment to remove my braces.  The doctor told me saying, “there was a lot of movement of your teeth, we probably need to install a permanent wire retainer behind your bottom front teeth”.  And at the same time I was told I needed to wear a retainer full time for 6 months, and then at night time for the rest of my life.

I was a little shocked as I never really put two and two together that after the braces I would have to wear a retainer at night in my mouth for the rest of my life.  I wonder if ortho offices gave a really honest assessment of the entire process (brackets breaking, wires poking, teeth sensitivity, retainers for the rest of your life), if fewer people would opt in.

Am I happy with the result?  Well my underbite is fixed now, but really the whole intended health outcome of being to chew when I’m 60 might require another blog post in 20ish years.

State of Data Engineering 2024 Q2

Data Engineering and AI

Chip Huyen, who came out of Stanford and is active in the AI space recently wrote an article on what she learned by looking at the 900 most popular open source AI tools.

Image Credit: huyenchip’s blog

In data engineering, one of our primary usages of AI is really just prompt engineering.

Use Case 1: Data Migration

Before LLMs, when we did data migrations, we would use Amazon Schema Conversion Tool (SCT) first to help convert source schemas to a new target schema.  Let us say we are going from SQLServer to Postgres, which is a major language change.

From there, the hard part begins where you need to manually convert the SQL Server SQL business logic code to Postgres.  Some converters do exist out there, and I assume they work on a basis of mapping a language grammar from one to another (fun fact – I almost pursued a PhD in compiler optimization, but bailed from the program).

Now what we can do is use LLMs to convert a huge set of code from one source to a target using prompt engineering.  Despite a lot of the new open source models out there, Chat GPT 4 still seems to be outperforming the competitors for the time being in doing this type of code conversions.

The crazy thing is with the LLMs, we can convert really one source system to any source system.  If you try it out Java to C#, SQL to Spark SQL, all work somewhat reasonably well.  In terms of predictions of our field I see a couple things progressing

Phase 1 Now:

  1. Productivity gains of code conversions using LLMs
  2. Productivity gains of coding itself of tools like Amazon Code Whisperer or Amazon Q or LLM of your choice for faster coding
  3. Productivity gains of learning a new language with LLMS
  4. Debugging stack traces by having LLMs analyze it

Phase 2: Near Future

  1. Tweaks of LLMs to make them more deterministic for prompt engineering.  We already have the ability to control creativity with the ‘temperature’ parameter, but we generally have to give really tight prompt conditions to get some of the code conversions to work.  In some of our experimentations with SQL to SparkSQL, doing things like passing in the DDLs have forced the LLMs to generate more accurate information.
  1. An interesting paper about using chain of thought with prompting (a series of intermediate reasoning steps), might help us move towards this
    Arxiv paper here –
  2. In’s latest newsletter, they mentioned a citation of a paper adding “Let’s think step by step” improved zero shot reasoning from 17 to 79%.  If you happen to DM me and say that in an introduction I will raise an eyebrow. citation link
  1. Being able to use LLMs to create data quality tests based on schemas or create unit tests based off existing ETL code.

Phase 3: Future

  1. The far scary future is where we tell LLMs how to create our data engineering systems.  Imagine telling it to ingest data from S3 into an Open Table Format (OTF) and to write business code on top of this.  I kind of don’t see this for at least 10ish years though.

Open Table Format Wars – Continued

The OTF wars continue to rage with no end in site.  As a refresher, there are 3 players

  1. Apache Hudi – which came out of the Uber project
  2. Apache Iceberg – which came out of the Netflix project
  3. Databricks Deltalake.

As a reminder, OTFs provide features some as time travel features, incremental ETL, deletion capability, and schema evolution-ish capability depending on which one you use.

Perhaps one of the biggest subtle changes which has recently happened is that the OneTable project is now Apache X Table.

Apache X Table is a framework to seamlessly do cross-table work between any of the OTFs.  I still think this is ahead of its time because I haven’t seen any project that have needs to combine multiple OTFs in an organization.  My prediction though is in 5-10 years this format will become a standard to allow vendor interoperability, but it will take a while.

Apache Hudi Updates

Apache Iceberg Updates

Lake Formation

Lake Formation, which still is a bit weird to me as one part of it is blue prints which we really don’t use, and the other part which deals with access control, rolled out some new changes with OTF integration and ACL

Summary about state of the OTF Market

It is still kind of mess, and there still really aren’t any clear winners.  There are also multiple options where you can choose to go the open source branch or with a hosted provide with One House or Tabular.

The false promises of AWS announcements – S3 Express Zones

Around Re:invent, there are always a huge set of announcements, and one stood out, S3 Express Zones.  This feature would allow retrieval of data in S3 in the single digit milliseconds with the tradeoffs of storage being in one zone (so no HA).  You can imagine if this actually works, datalakes can hypothetically start competing with databases as we wouldn’t need to worry about the SLA time penalty you usually get with S3.

Looking at the restrictions there are some pretty significant drawbacks.

As you can see here Hudi isn’t supported  (not sure why Iceberg tables aren’t there), and Deltalake has partial support.  The other consideration is this is in one zone, so you have to  make sure there is a replicated bucket in a standard zone.

I kind of feel that Amazon seems to test the waters by launching not fully formed products, to get feedback from us.  Unfortunately that makes us the guinea pigs

TLDR – This service works for Glue jobs, but for OTFs, it is dead in the water for the time being.

Amazon Q 

I remember being in an AWS roundtable representing data consulting companies at Re:invent and a complaint from other firms was that Amazon had too many confusing products.  As we are all guinea pigs in their early services, Amazon Q is no exception. 

 Use CaseFeatures
Amazon Q for BusinessChatbot for internal enterprise data that is managed by Amazon. No dev work requiredChatbot
Amazon Q For DevelopersBest for doing basic coding and coding with AWS specific services.

Broader coding is probably better with a foundational model
Code completion – Code whisperer

Chat – Amazon Q


  1. Amazon Q for business is a managed product where you click and add data sources and a chatbot is used
  2. Amazon Q for developers contains Code completion (Code Whisperer) AND a chat in Visual Studio IDE with, yes, Amazon Q again as the chat.  Confused yet?

Quicksight Q

I’d like to confuse you one more time with the history of Quicksight Q.  Pre ChatGPT and LLM craze, Quicksight Q in 2021 went Generally Available (GA) being powered by Machine Learning

After Chat GPT came out, Quicksight Q went back into Preview

With LLM integration, but they kept the same name.

One of the things to really keep in mind is as you do your solutions architecture, you need to keep in mind of a service is in preview or GA.  Things in preview typically only support a couple regions and don’t have production support.  If you are interested in a service in preview (like Amazon Q), it is advisable to wait a bit.

A Framework for Processing Uber Uber Large Sets of Data – Theseus

I show this diagram very often, and as a refresher, a lot of the work we do in data engineering is yellow and in red, and often involves OTFS.

Voltron Data, who created a GPU Query Engine called Theseus, put out these benchmarks comparing their framework Theseus vs Spark

Image Credit: Voltran’s Blog1
Their guidance also quite interesting

For less than 2TBs: We believe DuckDB and Arrow backed projects, DataFusion, and Polars make a lot of sense. This is probably the majority of datasets in the world and can be run most efficiently leveraging these state-of-the-art query systems.

For up to 30TBs: Well-known data warehouses like Snowflake, Google BigQuery, Databricks, and distributed processing frameworks like Spark and Trino work wonders at this scale.

For anything over 30TBs: This is where Theseus makes sense. Our minimum threshold to move forward requires 10TB queries (not datasets), but we prefer to operate when queries exceed 100TBs. This is an incredibly rare class of problem, but if you are feeling it, you know how quickly costs balloon, SLAs are missed, and tenuously the data pipeline is held together.

I mostly work in the AWS space, but it is interesting to peek on what innovations are going on outside of the space.

The author of Apache Arrow also made this observation

  • </= 1TB — DuckDB, Snowflake, DataFusion, Athena, Trino, Presto, etc.
  • 1–10TB — Spark, Dask, Ray, etc.
  • 10TB — hardware-accelerated processing (e.g., Theseus).
    (citation credit link)

You might ask, what my guidance might be for the Amazon space?

  • < 100 gigabytes – your run of the mill RDS or Aurora
  • >= 100 gigabytes – 30 TB – Redshift, or OTF
  • >30 TB – We haven’t really played in this space but things like Apache Iceberg are probably better candidates

TLDR – you probably will never use Theseus, so this is just a fun article.

American Privacy Rights Act (APRA)

There was a bit of surprising news coming out of the US Congress that there is now draft legislation for a national data privacy rights for Americans.  In the United States, data privacy has consisted of an odd patchwork of legislation state to state (like CCPA in California or the Colorado Privacy Act).  The US really is quite behind in legislation as the rest of the world has some type of privacy legislation.

Here are some draft highlights

  1. Deletion Requests: Companies are required to delete personal data upon an individual’s request and must notify any third parties who have received this data to do the same.
  2. Third-Party Notifications: Companies must inform third parties of any deletion requests, ensuring that these third parties also delete the relevant data.
  3. Verification of Requests: Companies need to verify the identity of individuals who request data deletion or correction to ensure the legitimacy of these requests.
  4. Exceptions to Deletion: There are specific conditions under which a company may refuse a deletion request, such as legal restrictions, implications for data security, or if it would affect the rights of others.
  5. Technological and Cost Constraints: If it is technologically impossible or prohibitively expensive to comply with a deletion request, companies may decline the request but must provide a detailed explanation to the individual.
  6. Frequency and Cost of Requests: Companies can allow individuals to exercise their deletion rights free of charge up to three times per year; additional requests may incur a reasonable fee.
  7. Timely Response: Companies must respond to deletion requests within specified time frames, generally within 15 to 30 days, depending on whether they qualify as large data holders or not.

Who is this applicable for?

  1. Large Data Holders: The Act defines a “large data holder” as a covered entity that, in the most recent calendar year, had annual gross revenue of not less than $250 million and, depending on the context, meets certain thresholds related to the volume of covered data processed. These thresholds include handling the covered data of more than 5 million individuals, 15 million portable connected devices identifying individuals, or 35 million connected devices that can be linked to individuals. Additionally, for handling sensitive covered data, the thresholds are more than 200,000 individuals, 300,000 portable connected devices, or 700,000 connected devices​​.
  2. Small Business Exemptions: The Act specifies exemptions for small businesses. A small business is defined based on its average annual gross revenues over the past three years not exceeding $40 million and not collecting, processing, retaining, or transferring the covered data of more than 200,000 individuals annually for purposes other than payment collection. Furthermore, all covered data for such purposes must be deleted or de-identified within 90 days unless retention is necessary for fraud investigations or consistent with a return or warranty policy. A small business also must not transfer covered data to a third party in exchange for revenue or other considerations​​.

A while back I worked on a data engineering project which was exposed to the European GDPR.  It was interesting because we had meetings with in-house counsel lawyers to discuss what kind of data policies they had in place.  One of the facets of GDPR which is similar here is the ‘right to remove data.’

We entered some gray areas as when talking with lawyers the debate was occurring which data would be removed?  Removing data from a database or data lake is clear if it contained customer data, but what if it was deeply nestled in Amazon Glacier?

I don’t really have any great answers, but if this legislation actually does pan out, it makes a strong case for large companies to use OTFs for their data lakes otherwise it would be extremely difficult to delete the data.

TLDR – if you are a solution architect, do ask what kind of data policy exposure they have.  If this legislation does pass, please pay attention when you start your projects based in the USA whether this legislation is applicable to them based of the final legislation.

Citation Link and Credit For Talking About This – Hard Fork Podcast


Everything Else

Glue: Observability

The AWS Team recently put out a blog series on monitoring and debugging AWS Jobs using observability metrics. (part 3)


  • The DBT team also released their 2024 state of analytics engineering (PDF here) –
  • TLDR, data quality is still of big concern
  • I’m surprised data mesh is still a thing, although it seems like it is only for big orgs according to the survey

AWS Exams:

AWS released a free training course on the Data Engineer Associate Exam

Also note the AWS Specialty Analytics and Database Specialty exams are being retired this month.

YADT (Yet Another Data Tool)
As if there weren’t enough tools on the market..


Fortunately uhh I don’t think anyone in our team is named Devon, but this video has been making its rounds the Internet as the first ‘AI software engineer’

Just remember, Devon hasn’t taken our jobs…. yet.

State of Data Engineering 2024 Q1

The current state of data engineering offers a plethora of options in the market, which can be challenging when selecting the right tool We are approaching a period where the traditional boundaries between between databases, datalakes, and data warehouses are overlapping. As always, it is important to think about what is the business case, then do a technology selection afterwards.

This diagram is simple, but merits some discussion.

Most companies in the small and medium data fields can get away with simpler architectures with a standard database powering their business applications. However it is when you get into big data and extremely large data do you want to start looking at more advanced platforms.

The Open Source Table Format Wars Revisited

A growing agreement is forming around the terminology used for Open Table Formats (OTF), also known as Open Source Table Formats (OSTF). These formats are particularly beneficial in scenarios involving big data or extremely large datasets, similar to those managed by companies like Uber and Netflix. Currently, there are three major contenders in the OTF space.

PlatformLinkPaid Provider
Apache Hudihttps://hudi.apache
Apache Iceberg
Databricks Via hyperscaler

Several announcements from AWS recently, lead me to believe of some more support of Apache Iceberg into the AWS ecosystem

AWS Glue Data Catalog now supports automatic compaction of Apache Iceberg tables

Every datalake eventually suffers from a small file problem. What this means is if you have too many files in a given S3 partition (aka file path), performance degrades substantially. To alleviate this, compaction jobs are run to merge files to bigger files to improve performance. In managed paid platforms, this is done automatically for you, but in the open source platforms, developers are on the hook in needing to do this.

I was surprised to read that now if you use Apache Iceberg tables, developers no long have to deal with compaction jobs. Now to the second announcement:

Amazon Redshift announces general availability of support for Apache Iceberg 

If you are using Amazon Redshift, you can do federated queries without needing to go through the hassle of manually mounting data catalogs.

In this video, you can watch Amazon talk about Iceberg explicitly in their AWS storage:session from re:Invent.

This generally leads me to believe that Apache Iceberg probably will be more integrated into the Amazon ecosystem in the near future.

Apache Hudi

Apache Hudi recently released version 0.14.0 which has some major changes such as Record Level Indexing

One Table 
Another kind of weird development which was announced right before Re:invent was the announcement of OneTable,

Microsoft, the Hudi team, and the Databricks team got together to create a new standard that serves as an abstraction layer on top of an OTF.  This is odd to me, because not too many organizations have these data stacks concurrently deployed.

However probably in the next couple years as abstraction layers get created on top of OTFs, this will be something to watch.

Amazon S3 Express One Zone Storage Class

Probably one of the most important but probably buried news from re:Invent was the announcement of Amazon S3 Express One Zone

With this, we can now have single digit millisecond access to data information to S3, which leads to a weird question of datalakes encroaching onto database territory if they now can meet higher SLAs. However there are some caveats with this as there is limited region availability, and it is in one zone so think about your disaster recovery requirements. This is one feature I would definitely watch.

Zero ETL Trends

Zero ETL is the ability for behind the scenes replication for Aurora, RDS, and Dynamo to replicate to Redshift.  If you have a use case where Slowly Changing Dimensions (SCD) Type 1 is acceptable, these are all worth taking a look at. From my understanding, when replication occurs, there is no connection penalty to your Redshift cluster. 
Amazon OpenSearch Service zero-ETL integration with Amazon S3 preview now available 
AWS announces Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service 
AWS CloudTrail Lake data now available for zero-ETL analysis in Amazon Athena

Spark/Glue/EMR Announcements

Glue Serverless Spark UI 

Now it is way easier to debug glue jobs as the Spark UI doesn’t have to manually be provisioned.

Glue native connectors: Teradata, SAP HANA, Azure SQL, Azure Cosmos DB, Vertica, and MongoDB

AWS Glue announces entity-level actions to manage sensitive data 

Glue now supports Gitlab and Bitbucket

Trusted identity propagation 

Propagate oauth 2.0 credentials to EMR


Announcing Amazon Aurora Limitless Database


It is exciting to see the OTF ecosystem evolve. Apache Hudi is still a great and mature option, with Apache Iceberg now being more integrated with the AWS ecosystem.

Zero ETL has the potential to save your organization a ton of time if your data sources are supported by it.

Something to consider is that major shifts in data engineering occur every couple of months, so keep an eye on new developments, as they can have profound impacts on enterprise data strategies and operations.

State of Data Engineering 2023 Q3

As we roll towards the end of the year data engineering as expected does have some changes, but now everyone wants to see how Generative AI intersects with everything. The fits are not completely natural, as Generative AI like Chat GPT is more NLP type systems, but there are a few interesting cases to keep an eye on. Also Apache Iceberg is one to watch now there is more first class Amazon integration.

Retrieval Augmented Generation (RAG) Pattern

One of the major use cases for data engineers to understand for Generative AI is the retrieval augmented generation (rag) pattern.

There are quite a few articles on the web articulating this such as

What is important to realize is that Generative AI is only providing the light weight wrapper interface to your system. The RAG paradigm was created to help address context limitations by vectorizing your document repository and using some type of nearest neighbors algorithm to find the relevant data and passing it back to a foundation model. Perhaps LLMS with newer and larger context windows (like 100k context) may address these problems.

At the end of the data engineers will be tasked more with chunking, and vectorizing back end systems, and debates probably will emerge in your organization whether you want to roll out your own solution or just use a SAAS to do it quickly.

Generative AI for Data Engineering?

One of the core problems with generative AI is eventually it will start hallucinating. I played around with asking ChatGPT to convert CSV to JSON, and it worked for about the first 5 prompts, but by the 6th prompt, it started to make up JSON fields which never existed. 
Things I kind of envision in the future is the ability to use LLMs to stitch parts of data pipelines concerning data mapping and processing. But at the moment, it is not possible because of this. 
There is some interesting research occurring where a team has put a finite state machine (FSM) with LLMs to create deterministic JSON output. I know that might not seem like a big deal, but if we can address deterministic outcomes of data generation, it might be interesting to look at

So far use cases we see day to day are 

1.      Engineers using LLMs to help create SQL or Spark code scaffolds 

2.      Creation of synthetic data – basically pass in a schema and ask an LLM to generate a data set for you to test 

3.      Conversion of one schema to another schema-ish. This kind of works, but buyer beware 

Apache Iceberg

Last year our organization did a proof of concept with Apache Iceberg, but one of the core problems, is that Athena and Glue didn’t have any native support, so it was difficult to do anything.

However on July 19, 2023 AWS quietly released an integration with Apache Iceberg & Athena into production 

Since then, AWS has finally started to treat Iceberg as a first class product with their documentation and resources 

Something to keep track of is that the team which founded Apache Iceberg, founded a company called which provides hosted compute for Apache Iceberg workloads. Their model is pretty interesting because what you do is give Tabular access to your S3 buckets and they will deal with ingestion, processing, and file compaction for you. They even can point to DMS CDC logs, and create SCD Type 1, and query SCD Type 2 via time travel via a couple clicks which is pretty fancy to me.

However if you choose to roll things out yourself, expect to handle engineering efforts similar to this

The Open Source Table Format Wars Continue

One of the core criticisms of traditional datalakes the difficulty to perform updates or deletes against them. With that, we have 3 major players in the market for transactional datalakes. 

PlatformLinkPaid Provider
Databricks Via hyperscaler
Apache Hudihttps://hudi.apache
Apache Iceberg

What’s the difference between these 3 you say? Well, 70% of the major features are similar, but there are some divergent features

Also don’t even consider AWS Governed Tables and focus on the top 3 if you have these use cases.

Redshift Serverless Updates 

There has been another major silent update that now Redshift Serverless only requires 8 RPUs to provision a cluster. Before it was 32 RPUs which was ridiculously high number

8 RPUs x 12 hours x 0.36 USD x 30.5 days in a month = 1,054.08 USD 

Redshift Serverless cost (monthly): 1,054.08 USD 

Ra3.xlplus – 1 node 

792.78 USD 

So as you can see provisioned is still cheaper, but look into Serverless if 

·         You know your processing time of the cluster will be 50% idle 

·         You don’t want to deal with the management headaches 

·         You don’t need a public endpoint 


Data Built Tool (dbt), has really been gaining a lot of popularity at the moment. It is kind of weird for this pendulum to be swinging back and forth as originally many years ago we had these super big SQL scripts running on data warehouses. That went out of fashion, but now here we are 

A super interesting thing that got released is a dbt-glue adapter.

That means you can now run dbt SQL processing on Athena now 

For those new to dbt feel free to check this out

Glue Docker Image

A kind of a weird thing, but I recently saw the ability to launch Glue as a local docker image. I haven’t personally tried this, but it is interesting

Zero ETL Intrigue

This is kind of an old feature, but Amazon rolled out in preview a Zero ETL method of MySQL 8.x to Redshift

This is pretty intriguing meaning SCD Type 1 views should be replicated without doing any work of putting data through a datalake. However it is still in preview, so I can’t recommend it until it goes into general release.

State of Data Engineering 2023 Q2

When looking at data engineering for your projects, it is important to think about market segmentation. In particular, you might be able to think about it in four segments

  • Small Data
  • Medium Data
  • Big Data
  • Lots and Lots of Data

Small Data – This refers to scenarios where companies have data problems (organization, modeling, normalization, etc), but don’t necessarily generate a ton of data. When you don’t have a lot of data, different tool sets are in use ranging from low code tools to simpler storage mechanisms like SQL databases.

Low Code Tools 

The market is saturated with low code tools, with an estimated 80-100 products available. Whether low code tools work for you depends on your use case. If your teams lack a strong engineering capacity, it makes sense to use a tool to help accomplish ETL tasks.

However, problems arise when customers need to do something outside the scope of the tool.

Medium Data– This refers to customers who have more data, making it sensible to leverage more powerful tools like Spark. There are several ways to solve the problem with data lakes, data warehouses, ETL, or reverse ETL.

Big Data – This is similar to medium data, but introduces the concepts of incremental ETL (aka transactional data lakes or lake houses). Customers in this space tend to have data in the hundreds gigabytes to terabytes.

Transactional data lakes are essential because incremental ETL is challenging. For example, consider an Uber ride to the airport that costs $30. Later, you give a $5 tip, and now your trip costs $35. In a traditional database, you can run some ETL to update the script. However, Uber has tons of transactions worldwide, and they need a different way of dealing with the problem.

Introducing transactional data lakes requires more operational overhead, which should be taken into consideration.

Lots and Lots of Data – Customers in this space generate terabytes or petabytes of data a day. For example, Walmart creates 10 pb of data (!) a day.

When customers are in this space, transactional data lakes with Apache Hudi, Apache Iceberg, and Databricks Deltalake are the main tools used.


The data space is large and crowded. With the small and lots of data sizes, the market segment is clear. However, the mid-market data space will probably take some time for winners to emerge.

Data Engineering Low Code Tools

In the data engineering space we have seen quite a few low code and no code tools pass through our radar. Low code tools have their own nuances as you will get to operationalize quicker, but the minute you need to customize something outside of the toolbox, you may run into problems. That’s when we usually deploy our custom development using things like Glue, EMR, or even transactional datalakes depending on your requirements.

This list is split into open source, ELT (reverse ETL), streaming, popular tools, and the rest of the tools. In the space, one thing I have been looking for is a first class open source product. I know that many of these products start as open source and end up releasing a managed version of the product. Personally of course I am all in for open source teams to make back their money somehow, but it would be ideal to have the platforms still contain an open source license.

One thing my team has been noticing is the traction dbt has been gaining in the market. It flips the paradigm a bit doing ELT (Extract Load Transform – reverse ETL), where everything is loaded to your data warehouse first then you start doing transformations on it.

Another project I have been watching with Zach Wilson’s recommendation is It is a pretty spiffy way of creating quick DAGs with executable Python notebooks. The platform is pretty active soliciting feedback on Slack and is one to watch for the future. Airbyte and Meltano are newer to me and I hope to take some time to play with those tools. This list is by no means the most exhaustive, but let me know if there is anything I have missed.

Opensource Tools

Product: Airbyte
Description: Airbyte is an open-source data integration platform that allows users to replicate data from various sources and load it into different destinations. Its features include real-time data sync, robust data transformations, and automatic schema migrations.
Github Link:
Cost: Free, with paid plans available
Release Date: 2020
Number of Employees: 11-50

Description: is a no-code AI platform that enables businesses to automate and optimize workflows. It includes features such as visual recognition, natural language processing, and predictive analytics, with a focus on e-commerce applications.
Github Link:
Cost: Open source
Release Date: 2020
Number of Employees: 11-50

Product: Meltano
Description: Meltano is an open-source data integration tool that allows users to build, run, and manage data pipelines using YAML configuration files. Its features include source and destination connectors, transformations, and orchestration.
Github Link:
Cost: Free, with paid options available
Release Date: 2020
Number of Employees: 11-50

Product: Apache Nifi
Description: Apache Nifi is a web-based dataflow system that allows users to automate the flow of data between systems. Its features include a drag-and-drop user interface, data provenance, and support for various data sources and destinations.
Github Link:
Cost: Free
Release Date: 2014
Number of Employees: N/A

Product: Apache Beam
Description: Apache Beam is an open-source, unified programming model for batch and streaming data processing. It provides a simple, portable API for defining and executing data processing pipelines, with support for various execution engines.
Github Link:
Cost: Free
Release Date: N/A
Number of Employees: N/A


Product: dbt (data build tool)
Description: dbt is an open-source data transformation and modeling tool that enables analysts and engineers to transform their data into actionable insights. It provides a simple, modular way to manage data transformation pipelines in SQL, with features such as version control, documentation generation, and testing.
Github Link:
Cost: Free, with paid options available for enterprise features and support
Release Date: 2016
Number of Employees: 51-200


Product: Confluent
Description: Confluent is a cloud-native event streaming platform based on Apache Kafka that enables organizations to process, analyze, and respond to data in real-time. It provides a unified platform for building event-driven applications, with features such as data integration, event processing, and management tools.
Github Link:
Cost: Free, with paid options available for enterprise features and support
Release Date: 2014
Number of Employees: 1001-5000

Popular Tools

Product: Fivetran
Description: Fivetran is a cloud-based data integration platform that automates the process of data pipeline building and maintenance. It provides pre-built connectors for over 150 data sources and destinations, with features such as data synchronization, transformation, and monitoring.
Github Link:
Cost: Subscription-based, with a free trial available
Release Date: 2012
Number of Employees: 501-1000

Product: Alteryx
Description: Alteryx is an end-to-end analytics platform that enables users to perform data blending, advanced analytics, and machine learning tasks. It provides a drag-and-drop interface for building and deploying analytics workflows, with features such as data profiling, data quality, and data governance.
Github Link:
Cost: Subscription-based, with a free trial available
Release Date: 1997
Number of Employees: 1001-5000

Product: Informatica
Description: Informatica is a data management platform that enables users to integrate, manage, and govern data across various sources and destinations. It provides a unified platform for data integration, quality, and governance, with features such as data profiling, data masking, and data lineage.
Github Link:
Cost: Subscription-based, with a free trial available
Release Date: 1993
Number of Employees: 5001-10,000

Product: Matillion
Description: Matillion is a cloud-native ETL platform that enables users to extract, transform, and load data into cloud data warehouses. It provides a visual interface for building and deploying ETL workflows, with features such as data transformation, data quality, and data orchestration.
Github Link:
Cost: Subscription-based, with a free trial available
Release Date: 2011
Number of Employees: 501-1000

Orchestration Tools

Sure! Here are the entries for Prefect, Dagster, Airflow, Azkaban, Luigi, and Oozie:

Product: Prefect
Description: Prefect is a modern data workflow orchestration platform that enables users to automate their data pipelines with Python. It provides a simple, Pythonic interface for defining and executing workflows, with features such as distributed execution, versioning, and monitoring.
Github Link:
Cost: Free, with paid options available for enterprise features and support
Release Date: 2018
Number of Employees: 51-200

Product: Dagster
Description: Dagster is a data orchestrator and data integration testing tool that enables users to build and deploy reliable data pipelines. It provides a Python-based API for defining and executing pipelines, with features such as type-checking, validation, and monitoring.
Github Link:
Cost: Free, with paid options available for enterprise features and support
Release Date: 2019
Number of Employees: 11-50

Product: Airflow
Description: Airflow is an open-source platform for creating, scheduling, and monitoring data workflows. It provides a Python-based API for defining and executing workflows, with features such as task dependencies, retries, and alerts.
Github Link:
Cost: Free
Release Date: 2015
Number of Employees: N/A (maintained by the Apache Software Foundation)

Product: Azkaban
Description: Azkaban is an open-source workflow manager that enables users to create and run workflows on Hadoop. It provides a web-based interface for creating and scheduling workflows, with features such as task dependencies, notifications, and retries.
Github Link:
Cost: Free
Release Date: 2010
Number of Employees: N/A (maintained by the Azkaban Project)

Product: Luigi
Description: Luigi is an open-source workflow management system that enables users to build complex pipelines of batch jobs. It provides a Python-based API for defining and executing workflows, with features such as task dependencies, retries, and notifications.
Github Link:
Cost: Free
Release Date: 2012
Number of Employees: N/A (maintained by Spotify)

Product: Oozie
Description: Oozie is a workflow scheduler system for managing Hadoop jobs. It provides a web-based interface for defining and scheduling workflows, with features such as task dependencies, triggers, and notifications.
Github Link:
Cost: Free
Release Date: 2009
Number of Employees: N/A (maintained by the Apache Software Foundation)


3forge – – 3forge delivers software tools for creating financial applications and data delivery platforms.

Ab Initio Software – – Ab Initio Software provides a data integration platform for building large-scale data processing applications.

Adeptia – – Adeptia offers a cloud-based, self-service integration solution that allows users to easily connect and automate data flows across multiple systems and applications.

Aera – – Aera provides an AI-powered platform for enterprises to accelerate their digital transformation by automating and optimizing business processes.

Aiven – – Aiven offers managed cloud services for open-source technologies such as Kafka, Cassandra, and Elasticsearch. – – provides a unified data platform that allows users to build, scale, and automate data pipelines across various sources and destinations.

Astera Software – – Astera Software offers a suite of data integration and management tools for businesses of all sizes.

Black Tiger – – Black Tiger provides an open-source data pipeline framework that simplifies the process of building and deploying data pipelines.

Bryte Systems – – Bryte Systems offers an AI-powered data platform that helps organizations manage their data operations more efficiently.

CData Software – – CData Software provides a suite of drivers and connectors for integrating with various data sources and APIs.

Census – – Census offers an automated data syncing platform that allows businesses to keep their customer data up-to-date across various systems and applications.

CloverDX – – CloverDX provides a data integration platform for building and managing complex data transformations.

Data Virtuality – – Data Virtuality offers a data integration platform that allows users to connect and query data from various sources using SQL.

Datameer – – Datameer provides a data preparation and exploration platform that enables users to analyze large datasets quickly and easily.

DBSync – – DBSync provides a cloud-based data integration platform for connecting and synchronizing data across various systems and applications.

Denodo – – Denodo provides a data virtualization platform that allows users to access and integrate data from various sources in real-time.

Devart – – Devart offers a suite of database tools and data connectivity solutions for various platforms and technologies.

DQLabs – – DQLabs provides a self-service data management platform that automates the process of discovering, curating, and governing data assets.

eQ Technologic – – eQ Technologic offers a data integration platform that enables users to extract, transform, and load data from various sources.

Equalum – – Equalum provides a real-time data ingestion and processing platform that enables organizations to make data-driven decisions faster.

Etleap – – Etleap offers a cloud-based data integration platform that simplifies the process of building and managing data pipelines.

Etlworks – – Etlworks provides a data integration platform that allows users to create and manage complex data transformations.

Harbr – – Harbr is a data exchange platform that connects and facilitates secure data collaboration between organizations.

HCL Technologies (Actian) – – Actian provides hybrid cloud data analytics software solutions that enable organizations to extract insights from big data and act on them in real time.

Hevo Data – – Hevo Data provides a cloud-based data integration platform that enables companies to move data from various sources to a data warehouse or other destination in real time.

Hitachi Vantara – – Hitachi Vantara provides data management, analytics, and storage solutions for businesses across various industries.

HULFT – – HULFT provides data integration and management solutions that enable businesses to streamline data transfer and reduce data integration costs.

ibi – – ibi provides data and analytics software solutions that help organizations make data-driven decisions.

Impetus Technologies – – Impetus Technologies provides data engineering and analytics solutions that enable businesses to extract insights from big data.

Infoworks – – Infoworks provides a cloud-native data engineering platform that automates the process of data ingestion, transformation, and orchestration.

insightsoftware – – insightsoftware provides financial reporting and enterprise performance management software solutions that help organizations improve their financial and operational performance. – – provides a cloud-based data integration platform that enables businesses to integrate and manage data from various sources.

Intenda – – Intenda provides a data integration and analytics platform that enables businesses to unlock insights from their data.

IRI – – IRI provides data management and integration software solutions that enable businesses to integrate and manage data from various sources.

Irion – – Irion provides a data management and governance platform that enables businesses to automate data quality and compliance processes.

K2view – – K2view provides a data fabric platform that enables businesses to connect and manage data across various sources and applications.

Komprise – – Komprise provides an intelligent data management platform that enables businesses to manage and optimize data across various storage tiers.

Minitab – – Minitab is a statistical software package designed for data analysis and quality improvement.

Nexla – – Nexla offers a data operations platform that automates the process of ingesting, transforming, and delivering data to various systems and applications.

OpenText – – OpenText is a Canadian company that provides enterprise information management software.

Palantir – – Palantir is an American software company that specializes in data analysis.

Precisely – – Precisely provides data integrity, data integration, and data quality software solutions.

Primeur – – Primeur is an Italian software company that offers products and services for data integration, managed file transfer, and digital transformation.

Progress – – Progress is an American software company that provides products for application development, data integration, and business intelligence.

PurpleCube – – PurpleCube is a Canadian consulting company that specializes in data integration, data warehousing, and business intelligence.

Push – – Push is a French software company that provides products and services for data processing and analysis.

Qlik – – Qlik provides business intelligence software that helps organizations visualize and analyze their data.

RELX (Adaptris) – – Adaptris, now a RELX company, offers data integration software that helps organizations connect systems and applications.

Rivery – – Rivery is a cloud-based data integration platform that allows businesses to consolidate, transform, and automate data.

Safe Software – – Safe Software provides spatial data integration and spatial data transformation software.

Semarchy – – Semarchy provides a master data management platform that helps organizations consolidate and manage their data.

Sesame Software – – Sesame Software offers data management solutions that simplify data integration, data warehousing, and data analytics.

SnapLogic – – SnapLogic provides a cloud-based integration platform that enables enterprises to connect cloud and on-premise applications and data.

Software AG – – Software AG offers a platform that enables enterprises to integrate and optimize their business processes and systems.

Stone Bond Technologies – – Stone Bond Technologies offers a platform that enables enterprises to integrate data from various sources and systems.

Stratio – – Stratio offers a platform that enables enterprises to process and analyze large volumes of data in real-time.

StreamSets – – StreamSets offers a data operations platform that enables enterprises to ingest, transform, and move data across systems and applications.

Striim – – Striim offers a real-time data integration and streaming analytics platform that enables enterprises to collect, process, and analyze data in real-time.

Suadeo – – Suadeo provides a platform that enables enterprises to integrate and manage their data from various sources.

Syniti – – Syniti offers a data management platform that enables enterprises to integrate, enrich, and govern their data.

Talend – – Talend provides a cloud-based data integration platform that enables enterprises to connect, cleanse, and transform their data.

Tengu – – Tengu offers a data engineering platform that enables enterprises to automate the process of ingesting, processing, and delivering data.

ThoughtSpot – – ThoughtSpot offers a cloud-based platform that enables enterprises to analyze their data in real-time.

TIBCO Software – – TIBCO Software offers a platform that enables enterprises to integrate and optimize their business processes and systems.

Tiger Technology – – Tiger Technology offers a platform that enables enterprises to manage, move, and share their data across systems and applications. – – provides a platform that enables enterprises to manage and process their data in real-time.

Upsolver – – Upsolver offers a cloud-native data integration platform that enables enterprises to process and analyze their data in real-time.

WANdisco – – WANdisco offers a platform that enables enterprises to replicate and migrate their data across hybrid and multi-cloud environments.

ZAP – – ZAP offers a data management platform that enables enterprises to integrate, visualize, and analyze their data.

Domo – – Domo is a cloud-native platform that gives data-driven teams real-time visibility into all the data and insights needed to drive business forward.

Dell Boomi – – Dell Boomi is a business unit acquired by Dell that specializes in cloud-based integration, API management, and Master Data Management.

Stitch – – Stitch is a cloud-first, open-source platform for rapidly moving data. It allows users to integrate with over 100 data sources and automate data movement to a cloud data warehouse.

Sparkflows – – Sparkflows is a low-code, drag-and-drop platform that enables organizations to build, deploy, and manage Big Data applications on Apache Spark.

Liquibase – – Liquibase is an open-source database-independent library for tracking, managing, and applying database schema changes.

Shipyard – – Shipyard is a container management platform that makes it easy to deploy, manage, and monitor Docker containers.

Flyway – – Flyway is an open-source database migration tool that allows developers to evolve their database schema easily and reliably across different environments.