English-language podcasts

The Databricks Data Engineer

Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders

13 min

Listen through the publisher, then discuss the episode here or arrange an in-person Roundtable. Your city determines where to meet.

Roundtables are designed for groups of four, with a shared subject and room for everyone to speak. Come for discussion, with no dating expectation.

Three schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer counts, and the head of finance asks which one is right. The trail ends at bronze, where rows that failed parsing were quietly dropped. The vendor keeps thirty days. Every label was there. What the medallion was invented to fix was not, and the pipeline had rebuilt the old mess across three schemas. In this episode: - What pipelines looked like before bronze, silver and gold, and the three pains the diagram was drawn to fix - Why dropping bad rows at bronze feels clean and quietly decides how far back you can rebuild - When skipping a layer is the right call, and why seven layers is not a safer bet than three - The question to ask about every gold table before your next architecture review This episode is for Databricks data engineers who build or inherit a medallion architecture and have wondered what silver is actually for. Whether you're designing a new lakehouse or maintaining one that looks textbook, you'll walk away with a way to check that each layer does a job no other layer does, and to defend it with more than "it's best practice." --- Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors. Follow The Databricks Data Engineer for new episodes every week. LinkedIn: linkedin.com/in/jrlasak Newsletter: dataengineer.wiki #DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Episode information is provided by the publisher. Listening takes place on the publisher’s service.

Listen at the source ↗

Featured places to meet in Palo Alto, CA

Explore these places and arrange a conversation through ThinkMatch.

City guide for Palo Alto, CA

More from The Databricks Data Engineer

The Databricks Data Engineer

Leave or stay? The 4 questions Databricks engineers keep asking (and the 10-minute test that answers them)

12 min

It's late. A recruiter message sits unanswered, and the question you keep circling is short: I'm bored. Is bored a good enough reason to leave? It almost never travels alone. Four questions about leaving keep coming up from Databricks data engineers, and only the wording changes. Is boredom enough? Is it too late to stay for the migration? What is a good stack worth against twenty thousand more? Am I running toward something, or just away? They sound like questions about salary, stack and timing. They're one question, and from the inside its two answers feel identical. In this episode: - Why boredom says more about the next six months than the last two years, and where to look before you act on it - How joining a Databricks migration late becomes either a strong senior interview story or an answer you can't give in a final round - What a good stack is actually worth against a bigger offer, and why the worse platform might have the higher ceiling - The question to ask in every interview

Discuss this episode

Community Poll

Schools should evaluate new teaching approaches by actual improvements in learning.

Share your view, then see how readers voted. No sign-in required. This poll is shared across stories.

Do you agree or disagree?

Choose one answer

Common Quandary

A Common Quandary to discuss

Another conversation to explore, separate from this news story.

Your coworker's religious holiday means you work the unpopular shift again. Is that fair?

A workplace accommodates religious observances by asking other employees to cover unpopular shifts. One coworker objects that nonreligious commitments can be equally meaningful but receive less formal protection.

Does equal respect require treating every kind of commitment identically?

Explore this quandary