🤖 This Week in AI & Data Engineering

AI and Data Engineering nuggets #edition31

What's on the list today?

• 🎨 AI Update - Gemini's image editing goes bananas

• 🐙 GitHub's MCP Server - Natural language Git interactions

• ⚡ Databricks LSQL - Lightweight SQL execution made simple

• 🧊 Iceberg V4 Proposal - Revolutionary single file commits

🎨 AI Update: Gemini's "Nano Banana" Image Editing Revolution

Google DeepMind just dropped a game-changing image editing model that's literally going bananas - they're calling it "Nano Banana" and it's now the top-rated image editing model globally according to LMARENA leaderboard. It is lightning fast.

What's New? 🚀

• Consistent character likeness - maintains the same person/pet across edits

• Multi-photo blending - combine multiple images seamlessly

• Multi-turn editing - iteratively refine images step by step

• Style transfer - apply textures and patterns between objects

Check it out for free at Google AI Studio

🐙 GitHub's MCP Server - The Future of AI-Git Integration

🔍 The GitHub MCP (Model Context Protocol) Server bridges the gap between AI tools and GitHub's platform. Instead of wrestling with REST or GraphQL APIs, you simply point your MCP-compatible client to the server and make natural language requests.

How it Works 🛠️ The magic happens when you use conversational language that gets automatically converted into structured, semantic API calls:

• "List all open issues in the data-pipeline repository"

• "Show me pull requests waiting for review"

• "Fetch metadata about the main.py file"

• "Create an issue for the bug we discussed"

Key Benefits ✨

• Universal compatibility: Works with Copilot Workspace, VS Code plugins, custom chat UIs, and any MCP-compatible host

• Natural interaction: No more memorizing API endpoints or crafting complex queries

• Standardized interface: One protocol for all your GitHub automation needs

• Real-time data: Direct access to live repository information

Getting Started Quick Setup:

{
  "servers": {
    "github": {
      "type": "http", 
      "url": "https://api.githubcopilot.com/mcp/"
    }
  }
}

Create /vscode/mcp.json in your project root, paste the config, complete OAuth flow, and you're ready to go!

⚡ Databricks LSQL - Lightweight SQL Made Simple

The Problem 🎯 Traditional Databricks SQL Connector comes with heavy dependencies and overhead - not ideal for serverless functions, containerized apps, or quick automation scripts.

The Solution: LSQL ⚡ Databricks LSQL (Lightweight SQL) provides stateless SQL execution with minimal dependencies, perfect for AWS Lambda, Azure Functions, or any scenario where fast startup matters.

Four Core Methods:

• fetch_all() - Iterator over all results

• fetch_one() - Single record retrieval

• fetch_value() - Single value extraction

• execute() - Query execution without results

Quick Setup:

# Install Databricks CLI and configure workspace with personal access token(https://docs.databricks.com/aws/en/dev-tools/cli/install)
databricks configure --token --profile <<profile_name>>

# Install LSQL
pip install databricks-labs-lsql

Real-World Example:

from databricks.sdk import WorkspaceClient
from databricks.labs.lsql.core import StatementExecutionExt
import json

# Setup connection using configured profile
w = WorkspaceClient(profile="dev")
see = StatementExecutionExt(w)

# Quick count query
count = see.fetch_value("SELECT COUNT(*) FROM samples.nyctaxi.trips")
print(f"Total records: {count}")

# Detailed table metadata
data = see.fetch_all("DESCRIBE DETAIL samples.nyctaxi.trips")
table_info = [row.asDict() for row in data]
print(json.dumps(table_info, default=str, indent=2))

Output

[
  {
    "format": "delta",
    "id": "dee488f1-3017-49da-83f5-4c846ea845e9",
    "name": "samples.nyctaxi.trips",
    "description": null,
    "location": "abfss://metastore@ucstprdwesteu.dfs.core.windows.net/17a8f892-3592-4cda-a60f-4dd7892dc6fe/tables/1a254ba2-c40b-4707-b05c-46de6c121156",
    "createdAt": "2025-08-28 15:06:12.064000+00:00",
    "lastModified": "2025-08-28 15:06:19+00:00",
    "partitionColumns": [],
    "clusteringColumns": [],
    "numFiles": 1,
    "sizeInBytes": 456546,
    "properties": {
      "delta.dropFeatureTruncateHistory.retentionDuration": "0 hours",
      "delta.enableDeletionVectors": "false"
    },
    "minReaderVersion": 1,
    "minWriterVersion": 1,
    "tableFeatures": [],
    "statistics": {},
    "clusterByAuto": false
  }
]

When to Choose LSQL vs Traditional Connector:

• Use LSQL for: Serverless apps, automation scripts, quick queries, minimal overhead scenarios

• Use SQL Connector for: Heavy data transfers (GB+), complex cursor operations, ultra-low latency requirements

🧊 Iceberg V4 - The Single File Commit Revolution

The Problem 😤 Current Iceberg commits require writing at least 3 files (metadata.json, manifest list, manifest file), making small writes inefficient and causing metadata changes proportional to table size rather than operation size.

V4 Solution 🚀 The proposed V4 format introduces a "Root Manifest" that replaces manifest lists entirely. Key improvements:

• Single file writes for small operations

• Proportional scaling - metadata changes match operation size, not table size

• Better caching - manifests aren't constantly replaced

• Aggregate metrics at root level

Impact for Engineers 💪 This will dramatically improve streaming workload efficiency, small batch performance, and cost optimization for frequent updates - making Iceberg truly optimized for real-time analytics.