top of page

Search this site

59 results found with an empty search

  • Why Agent Evals Are the Most Underrated Part of AI Development

    You can have the most capable model, a well-engineered harness, and a solid product vision - and still have no idea if your agent is actually working. That's the problem evals solve. An evaluation ("eval") is a test for an AI system: give an AI an input, then apply grading logic to its output to measure success. Good evaluations help teams ship AI agents more confidently. Without them, it's easy to get stuck in reactive loops - catching issues only in production, where fixing one failure creates others. Why Agents Are Harder to Evaluate Than Chatbots A single-turn chatbot eval is simple: prompt in, response out, check if it's correct. Agents break this model entirely. Agents use tools across many turns, modifying state in the environment and adapting as they go - which means mistakes can propagate and compound. Frontier models can also find creative solutions that surpass the limits of static evals. For instance, Opus 4.5 solved a τ2-bench flight-booking problem by discovering a loophole in the policy. It "failed" the evaluation as written, but actually came up with a better solution for the user. This is why Anthropic draws a sharp distinction between transcript and outcome: a flight-booking agent might say "Your flight has been booked" at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database. Grade the outcome, not what the agent claimed. And remember: when you evaluate "an agent," you're evaluating the harness and the model working together - not the base model in isolation. The Build-Without-Evals Trap Most teams start with manual testing and gut feel. The breaking point comes when users report the agent feels worse after changes, and the team is flying blind. Absent evals, debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed. When more powerful models come out, teams without evals face weeks of testing while competitors with evals can upgrade in days. There's also a clarity dividend: two engineers reading the same initial spec could come away with different interpretations on how the AI should handle edge cases. An eval suite resolves this ambiguity. Three Types of Graders Agent evaluations typically combine three types of graders - code-based, model-based, and human - each evaluating some portion of either the transcript or the outcome. Code-based graders (regex, pass/fail, static analysis, tool call checks) are fast, cheap, and objective - but brittle to valid variations the designer didn't anticipate. LLM-as-judge is the most broadly applicable: these evaluators can be reference-free, allowing you to judge responses objectively without requiring ground truth answers. The trade-off is cost and non-determinism - they require calibration against human judgment to stay reliable. Human graders are the gold standard for quality and for calibrating automated graders, but don't scale. Automated evaluators can drift or fail in unexpected ways - keeping them aligned with real human perspectives takes constant upkeep. Capability vs. Regression Evals Capability evals ask "What can this agent do well?" - they should start at a low pass rate, targeting tasks the agent struggles with. Regression evals ask "Does the agent still handle everything it used to?" and should hold a nearly 100% pass rate. As teams hill-climb on capability evals, regression evals protect against backsliding. Descript runs two separate suites for quality benchmarking and regression. The Bolt AI team built an eval system with static analysis, browser agents, and LLM judges - all within three months of already having a widely-used product. Offline vs. Online Evaluation Offline evaluation runs your model on curated datasets - in CI pipelines or local dev tests - to catch regressions before they reach users. Online evaluation in a live environment lets you spot model drift or unexpected queries you never anticipated. A balanced approach - regular offline benchmarking plus continuous production monitoring - tends to yield the most robust results. LangSmith integrates evals directly into GitHub Actions so pipelines fail automatically when scores drop. Langfuse's execution tracing lets you inspect the exact prompt and response behind every LLM-as-judge score - so you can debug the grader, not just the agent. How Many Tasks Do You Need? You only need a handful of high-quality data points to get started. The quality and diversity of the data you're evaluating over directly influences how well the evaluation reflects real-world usage. Anthropic recommends 20–50 tasks drawn from real failures. Collect your hardest cases first - happy path tests tell you what works, failure cases tell you where to improve. The Bottom Line Without evals, it can be very difficult and time-intensive to understand how different model versions affect your use case. The ecosystem is mature enough that no team needs to start from scratch - Anthropic's eval framework, LangChain's OpenEvals and AgentEvals, Langfuse's observability platform, and OpenAI's Evals API all provide real infrastructure to build on. Evals aren't overhead. They're the difference between guessing your agent works and knowing it does. Anthropic Engineering \ Demystifying evals for AI agents LangChain Blog \ Quickly Start Evaluating LLMs with OpenEvals + AgentEvals LangSmith \ Evaluation Platform Langfuse \ LLM Evaluation 101 + LLM-as-a-Judge Tracing OpenAI \ Working with Evals OpenAI \ Evaluation Best Practices

  • Copilot vs Claude Code: Two Very Different Bets on How AI Should Understand Your Codebase

    Every AI coding tool faces the same fundamental problem: your codebase is too large to fit in a context window, but the model needs to understand it to help you. How a tool solves that problem shapes everything about what it can and can't do. GitHub Copilot and Claude Code solve it in opposite ways - and the gap between them isn't a matter of polish or features. It's a genuine architectural disagreement. How Copilot Works: Index First, Retrieve Later If your repository is hosted on GitHub or Azure DevOps, Copilot creates a remote index of your codebase and computes an embedding that captures patterns and relationships in your code. Based on your prompt, Copilot performs a semantic search on that index. Unlike traditional search, which matches exact words, semantic search focuses on meaning - using advanced vector embeddings, Copilot identifies files with the highest semantic similarity to your request and adds them to its context. In September 2025, GitHub shipped a new embedding model to power this retrieval layer - delivering a 37.6% lift in retrieval quality, about 2x higher throughput, and an 8x smaller index size. These vector representations retrieve semantically relevant code even when the exact words do not match. The architecture has layers. When eligible, Copilot executes two searches in parallel - the remote search hits GitHub's embeddings API, receiving semantically ranked chunks from the indexed commit, while a local diff search handles files modified since the last commit. For diffs between 301–2,000 files, it skips embeddings entirely and uses TF-IDF. Unchanged files use GitHub's fast indexed results; modified files use current code state. It's a sophisticated system, purpose-built for speed at scale. How Claude Code Works: Explore Now, Index Never Claude Code does none of this. There is no index, no embeddings, no pre-built vector database. Claude Code uses filesystem tools - Glob for file pattern matching, Grep for content search, and Read for loading specific files - to explore code on demand as it works through each task. Anthropic calls this "agentic search." The model decides what to look at. It reads the file tree, follows imports, searches for symbols, opens configs - the same way a new engineer would orient themselves in an unfamiliar codebase. The intelligence is in the model, not the retrieval infrastructure. This approach was inspired by how Boris Cherny observed engineers at Instagram searching code when the click-to-definition functionality in the in-house coding editor was broken. They used grep. It worked. Why Claude Code Abandoned RAG - In Boris's Own Words This wasn't a philosophical starting position. The team built RAG first and changed their minds based on results. Boris Cherny, creator of Claude Code, on X: "Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn't have the same issues around security, privacy, staleness, and reliability." On the Latent Space podcast, he went further: "We tried RAG… we tried a few different kinds of search tools. And eventually, we landed on just agentic search as the way to do stuff. One: it outperformed everything. By a lot. And this was surprising." The team tried several approaches - local vector databases, recursive model-based indexing, and others. All had downsides: stale indexes, permission complexity. Plain glob and grep, driven by the model, beat everything. The four reasons Boris cited for abandoning RAG: - Precision - grep finds exact matches; embeddings introduce fuzzy positives. In code, `createD1HttpClient` either appears in a file or it doesn't. Semantic similarity without textual match is usually noise. - Simplicity - no index to build, maintain, or sync. - Freshness - a pre-built index drifts from code the moment you start editing. Filesystem reads are always current. - Security & privacy - the index has to live somewhere. If that provider gets hacked, it's a liability. Even for Anthropic's own codebase, they don't want to upload it to a third-party system. Agentic search sidesteps all of that. The Real Trade-offs Neither approach is strictly better. They're optimized for different things. Copilot's index wins on: Speed of first retrieval. Semantic search across a massive, stable codebase - finding "where is authentication handled?" without knowing what it's called. Near-zero per-task token cost for retrieval. Claude Code's agentic search wins on: Precision for exact-match lookups. Always-current results with no staleness. Multi-step reasoning, where the model reads one file, learns something, and decides what to look at next. Zero setup, zero infrastructure. And critically - it gets better for free as models improve, because the intelligence stays in the model. The cost is real: every file Claude Code reads consumes tokens and takes time. On a large, stable codebase, a RAG system can surface relevant files in milliseconds for near-zero cost. Agentic search pays that cost on every task. What This Tells You About Each Tool's Design Philosophy Cherny describes Claude Code as "a Unix utility, not a product." No GUI, no managed index, no infrastructure - just a model with filesystem tools, composable into any workflow. The design bet is that model capability is the bottleneck, not retrieval speed, and that the model should be the one doing the reasoning. Copilot's bet is the opposite: invest in retrieval infrastructure so the model gets better inputs faster. The original framing from GitHub Next: "Real programming tasks are seldom self-contained. What you're about to write in one file crucially depends on details elsewhere in the codebase." Their answer is to build the system that surfaces those details before the model even starts. Both bets are coherent. They reflect different views of where the constraint actually lives - and as context windows grow and models improve, the answer will keep shifting. 1. Boris Cherny (X / Twitter) \ RAG vs Agentic Search 2. Boris Cherny (Latent Space Podcast, May 2025) \ Building Claude Code 3. Pragmatic Engineer \ Building Claude Code with Boris Cherny 4. GitHub Blog \ Copilot's new embedding model

  • Recursive Language Models: The Model That Calls Itself

    You've probably noticed something when using AI coding agents like Claude Code, Cursor, or Codex. The model doesn't just answer - it works. It reads a file, writes some code to search through it, reads another file, runs a check, and slowly pieces together an answer. It feels almost like watching someone think out loud. That intuition is important. Because a paper out of MIT CSAIL late last year - quietly titled Recursive Language Models - essentially formalizes exactly that behavior, pushes it to an extreme, and shows why it might be the key to handling truly massive contexts. The Problem: Context Rot Modern LLMs have large context windows - some handle hundreds of thousands of tokens. But here's the dirty secret: just because a model can read a million tokens doesn't mean it does so well. Feed a model a huge codebase or a long document and you'll notice it starts losing track of things mentioned early on, making inconsistent references, or just producing worse output overall. The researchers call this context rot - the gradual degradation in model quality as the input grows. It's not a hardware problem or a memory limit. It's a fundamental attention problem. The model is trying to hold too much in its head at once. The obvious solution - summarizing old context to make room for new stuff - has a fatal flaw: you lose information. Some details that seem unimportant early on turn out to matter later. Compaction is a lossy bet. The RLM Idea: Context as Environment, Not Input Here's the conceptual shift at the heart of RLMs. Normal LLMs treat context as data to be ingested - you pour the document in, the model reads it all, and produces an answer. The context lives inside the model's attention. RLMs treat context as an environment to be explored - the document lives outside the model, stored as a Python variable in a REPL. The model doesn't read it all upfront. Instead, it writes code to navigate it on demand. That `llm()` call? It's just a Python function in the REPL environment - one that happens to invoke the same model again. This is the "recursive" part. The model can call itself on smaller pieces, get results back as variables, and combine them into a final answer. It's divide and conquer, but the model is doing the dividing. How It Actually Works (Without the Magic) Let's be precise about something, because it's easy to get mystical here: the model isn't literally calling itself. There's no self-awareness involved. What's actually happening: The model generates text - specifically, Python code The REPL executes that code That execution triggers a new API call to the same model The result comes back as a Python variable The model sees the result and keeps generating The recursion lives entirely in the scaffolding - the harness around the model - not inside the model itself. The model is always doing one thing: predicting the next token. The infrastructure makes that feel like agency. This is a crucial point. RLMs aren't a new architecture. No new weights, no new attention mechanisms. It's a new inference strategy - a different way of orchestrating calls to an existing model. So How Is This Different From Existing Agents? This is where it gets interesting. When coding agents like Claude Code, Cursor, or Codex work through your codebase, they also navigate context on demand. They read files, write code, use tools, and build up an answer incrementally. From the outside, it looks a lot like what RLMs describe. But there's a structural difference in where the model sits: In typical agents, the model sits outside the environment. It's the orchestrator - given tools, files, and context, it directs operations from above. The model is the conductor of the orchestra. In RLMs, the model sits inside the environment. It's just another callable within the REPL. The document is a variable in that environment, and the model navigates from within. The model is an actor inside the play — one that can summon copies of itself. Fetches context on demand Writes code to navigate Explicit self-recursion Typical Yes Yes No (implicit) RLMs Yes Yes Yes (by design) The honest take: the line is thin. Most capable agents already do a lot of what RLMs describe, just without the formal self-recursion and without the explicit framing of context-as-variable. RLMs are less "here's something nobody does" and more "here's a rigorous formalization of something agents already do naturally - taken to its logical extreme." Why the Framing Matters Even if RLMs aren't radically new in practice, the framing is valuable. Most agent architectures today are built around the model as the top-level orchestrator. The model calls tools, tools return results, model continues. This works well for short-to-medium tasks. But as tasks get longer - multi-file codebases, lengthy legal documents, week-long agent trajectories - the orchestrator-on-top model starts to strain. The model still needs to hold a growing working memory of what's been done, what's been found, what still needs doing. RLMs suggest a different mental model: let the inference itself be recursive. Don't try to maintain a growing context at the top level. Instead, decompose the problem, dispatch sub-calls, and merge results. The context never has to get huge in any single call. The MIT paper shows this works - RLMs handle inputs two orders of magnitude beyond a model's context window, while maintaining quality that degrades sharply for vanilla frontier models. On GPT-5, the RLM approach outperformed the base model by a median of 26% against compaction strategies. What This Points To The broader trend here is a shift away from "stuff everything into context and hope" toward "let the model actively navigate and pull what it needs." RAG was an early version of this. Agent tool use was a further step. RLMs push it further still — making the model a first-class citizen of its own inference loop. There's an even more ambitious version of this idea: what if models were trained with RLM scaffolding, not just prompted into it? A model that has learned, through reinforcement, how to optimally decompose and recurse over long contexts would be qualitatively different from one just prompted to do so. The MIT team fine-tuned a small model (Qwen3-8B) around the RLM paradigm and saw it approach GPT-5 quality on long-context tasks. That's a hint at what's possible. RLMs aren't a new transformer. They're not a new architecture at all. They're a new answer to an old question: how should a model interact with more information than it can hold at once? The answer turns out to be: the same way a good engineer does. Don't read everything. Navigate intelligently. Break big problems into small ones. And don't be afraid to call in help - even if that help is yourself. Based on "Recursive Language Models" by Alex L. Zhang, Tim Kraska, and Omar Khattab - MIT CSAIL, arXiv:2512.24601 Original blog post by the authors: https://alexzhang13.github.io/blog/2025/rlm/

  • Bitcoin Taproot Explained: What Is Taproot?

    If you have been in the Bitcoin space or ecosystem for some time you all might have heard about taproot. But have you tried understanding it, if not then this article is for you. Let’s dive in, The Bitcoin network has undergone one of the biggest upgrades in its history. The Taproot is a most significant Bitcoin upgrade and as we all know every upgrade brings new features, privacy, security, and scalability to the software it is the same with Bitcoin. The main objective of Taproot is to make Bitcoin faster, more efficient, and more private. The upgrade brings a higher level of privacy, security, and scalability to the network. It reduces the storage of transaction data on Bitcoin’s network which allows Bitcoin to accommodate complex smart contracts and also makes it more scalable. Taproot components Schnorr Signatures MAST (Merklized Alternative Script Tree) Pay-2-Taproot (P2TR) If I want to keep it in simple terms, Taproot is an upgrade to the Bitcoin blockchain which improved key elements of its functionality. FYI, The last upgrade to Bitcoin was SegWit in 2017. https://bitcoin.org/en/

  • GraphQL Is More Than an Alternative to REST

    For years, REST has been the default language of web APIs. You create an endpoint, send a request, receive a response, and move on. That model works remarkably well until an application becomes complicated. Imagine opening an e-commerce application. The page needs your profile, recent orders, product information, inventory, reviews, recommendations, and delivery status. With a traditional REST architecture, the frontend may need to call several endpoints just to render one screen. GraphQL approaches the problem differently. Instead of asking the server for a predefined response, the client describes exactly what it needs. query { user(id: "42") { name email orders { id status total } } } The server responds with a structure that mirrors the query. { "data": { "user": { "name": "Name", "email": "name@example.com", "orders": [ { "id": "1001", "status": "SHIPPED", "total": 129.99 } ] } } } That simple idea is what made GraphQL so attractive to frontend teams. But in 2026, the interesting GraphQL conversation is no longer about whether a client can query fields. The interesting question is: How do you build a GraphQL API that remains fast, secure, observable, and maintainable when thousands of clients and multiple backend services depend on it? That is where GraphQL gets interesting. GraphQL Is More Than an Alternative to REST GraphQL is often described as a replacement for REST. I think that's the wrong way to look at it. REST and GraphQL solve different problems and can happily coexist in the same architecture. REST works particularly well when your API operations map naturally to resources and HTTP semantics. GraphQL becomes compelling when consumers need flexible access to connected data. Consider a banking application. A dashboard might need: query { account(id: "123") { balance owner { name } transactions(first: 20) { id amount timestamp } } } The client isn't really interested in the fact that this information may live in several backend systems. It wants one coherent view of the account. That is the fundamental strength of GraphQL: it can provide a graph-shaped interface over systems that are not naturally organized as a graph. Your database might be relational. Your payments service might expose REST. Your identity service might use another protocol. Your inventory system might be running as a separate microservice. GraphQL can sit above those systems and present a unified contract to consumers. Start With the Schema The best GraphQL projects don't begin by writing resolvers. They begin with the schema. A schema describes what clients are allowed to ask for. For example: type User { id: ID! name: String! email: String! orders: [Order!]! } type Order { id: ID! total: Float! status: OrderStatus! } enum OrderStatus { PENDING PAID SHIPPED CANCELLED } type Query { user(id: ID!): User } This immediately gives us a contract. A User has an ID, name, email, and orders. An Order has an ID, total, and status. The exclamation mark is significant. name: String! means the field is non-null. GraphQL's type system is one of the reasons the technology is so attractive for large applications. Instead of forcing frontend developers to interpret loosely structured JSON, the schema describes the available API surface explicitly. That schema becomes documentation, validation logic, tooling metadata, and a contract between teams. And that makes schema design one of the most important architectural decisions you'll make. Building the First GraphQL Server Let's make this practical. For our example, we'll use Node.js and the GraphQL.js ecosystem. Create a project: mkdir graphql-production cd graphql-production npm init -y npm install graphql graphql-http We can start with a very small schema, e.g. schema.js: import { buildSchema } from "graphql"; export const schema = buildSchema(` type User { id: ID! name: String! email: String! } type Query { user(id: ID!): User } `); For the moment, we'll keep the data in memory: const users = [ { id: "1", name: "Alice", email: "alice@example.com" }, { id: "2", name: "Bob", email: "bob@example.com" } ]; The resolver connects the schema to the actual data: const root = { user({ id }) { return users.find(user => user.id === id); } }; Complete schema.js file import { buildSchema } from "graphql"; export const schema = buildSchema(` type User { id: ID! name: String! email: String! } type Query { user(id: ID!): User } `); const users = [ { id: "1", name: "Alice", email: "alice@example.com" }, { id: "2", name: "Bob", email: "bob@example.com" } ]; export const root = { user({ id }) { return users.find(user => user.id === id); } }; Then create a small server file, e.g. server.js, to actually execute a query against it: import { graphql } from "graphql"; import { schema, root } from "./schema.js"; const query = ` query { user(id: "1") { id name email } } `; const result = await graphql({ schema, source: query, rootValue: root, }); console.log(JSON.stringify(result, null, 2)); If you SyntaxError: Cannot use import statement outside a module then enable enable ES modules in your package.json. { "name": "graphql-production", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "type": "module", "dependencies": { "graphql": "^17.0.2", "graphql-http": "^1.23.0" } } Now run node server.js You should get: { "data": { "user": { "id": "1", "name": "Alice", "email": "alice@example.com" } } } Now let's test GraphQL validation by changing the query to request a field that doesn't exist import { graphql } from "graphql"; import { schema, root } from "./schema.js"; const query = ` query { user(id: "1") { id name email age } } `; const result = await graphql({ schema, source: query, rootValue: root, }); console.log(JSON.stringify(result, null, 2)); Run: node server.js You should get: { "errors": [ { "message": "Cannot query field \"age\" on type \"User\". Did you mean \"name\"?", "locations": [ { "line": 7, "column": 7 } ] } ] } On the client-side you can send: query { user(id: "1") { id name email } } And the GraphQL runtime takes care of validating the query against the schema and executing the appropriate resolver. At this point, GraphQL looks almost deceptively simple. The real problems haven't arrived yet. The Resolver Problem Resolvers are where many GraphQL applications begin to suffer. Suppose we extend the schema: type User { id: ID! name: String! orders: [Order!]! } We might write: const resolvers = { User: { orders(user) { return database.orders.findMany({ userId: user.id }); } } }; Looks perfectly reasonable. Now imagine a query requesting 1,000 users: query { users { id name orders { id total } } } The GraphQL query itself isn't the problem. The problem is what happens underneath it. If the resolver executes one database query for every user, you can suddenly have something resembling: 1 query to retrieve users + 1,000 queries to retrieve orders = 1,001 database queries This is the famous N+1 problem. GraphQL's flexibility can therefore expose performance problems that aren't obvious from looking at the API request itself. DataLoader and the N+1 Problem A common solution is batching. Instead of asking the database separately for every user, we collect the requested IDs and perform a single operation. Conceptually: DataLoader A simplified implementation might look like: const userLoader = new DataLoader(async (userIds) => { const orders = await database.orders.findMany({ userId: { in: userIds } }); return userIds.map(id => orders.filter(order => order.userId === id) ); }); Now the application has an opportunity to turn hundreds or thousands of individual operations into a smaller number of batched operations. This is an important lesson in GraphQL development The shape of the GraphQL query does not tell you the cost of executing it. Two queries that look similar at the API layer can have dramatically different consequences inside the system. GraphQL Security Starts With the Query This is where GraphQL becomes particularly interesting from a security perspective. In a REST application, an attacker might repeatedly call: GET /users/123 With GraphQL, the attacker can construct the operation itself. For example: query { users { orders { items { product { reviews { author { orders { items { product { reviews { author { name } } } } } } } } } } } } The query may be syntactically valid. But the amount of work required to execute it could be enormous. This is why GraphQL security isn't simply about authentication. You need to control what queries clients are allowed to execute and how expensive those queries can become. Query Depth Is Only the Beginning A straightforward defense is query-depth limiting. If your application doesn't require queries deeper than eight levels, you can reject anything beyond that boundary. Conceptually: Depth 1 → allowed Depth 2 → allowed Depth 3 → allowed ... Depth 8 → allowed Depth 9 → rejected But depth alone isn't enough. Consider these two operations: users(first: 10) { name } and: users(first: 10000) { orders { items { product { reviews { author { name } } } } } } Their depth might not be dramatically different. Their computational cost certainly is. That leads us to query complexity. Instead of asking only: How deep is this query? we should ask: How expensive is this query likely to be? GraphQL's security guidance explicitly addresses depth limiting, query complexity, resource exhaustion, and other protections for production deployments. Complexity Limits Turn Queries Into Budgets Imagine assigning a cost to different fields. user = 1 orders = 5 items = 3 reviews = 10 The server can calculate an estimated cost before executing the operation. If the application's maximum allowed cost is 1,000, then: Cost 450 → execute Cost 900 → execute Cost 1,500 → reject This turns query execution into something resembling a resource budget. That's a much more useful security model than simply saying "GraphQL is typed, so it's safe." It isn't. The query itself is user-controlled input. Treat it accordingly. Authentication Isn't Authorization Another common mistake is assuming that a valid login means a GraphQL request is safe. Authentication answers: Who are you? Authorization answers: What are you allowed to do? Those are completely different questions. Imagine an authenticated employee requesting: query { customer(id: "123") { name email internalNotes } } Being authenticated doesn't mean that employee should automatically receive internalNotes. Authorization needs to happen at the appropriate layer of the application. A simplified resolver might look like: function resolveCustomer(_, args, context) { if (!context.user) { throw new Error("Unauthorized"); } if (!canReadCustomer(context.user, args.id)) { throw new Error("Forbidden"); } return getCustomer(args.id); } For larger systems, authorization is usually better centralized around domain policies rather than duplicated across hundreds of resolvers. The important principle is that the schema defines capability, but authorization determines whether a particular caller can exercise that capability. Don't Turn Your Database Into Your Schema GraphQL makes it tempting to expose database models directly. Suppose your database contains: Customer ├── password_hash ├── payment_token ├── internal_notes ├── risk_score └── admin_flags It doesn't mean your GraphQL type should contain all of those fields. A public API model should represent the contract you want consumers to use. For example: type Customer { id: ID! name: String! profile: Profile orders: [Order!]! } The GraphQL schema is an API boundary. Treat it like one. Pagination Isn't Optional A query such as: query { users { id name } } looks innocent. It becomes a problem when the database contains 20 million users. Production GraphQL APIs should put explicit boundaries around collections. Cursor-based pagination is a common approach: query { users(first: 20, after: "cursor") { edges { node { id name } } pageInfo { hasNextPage endCursor } } } Now the server controls how much data can be retrieved at once. Pagination isn't merely a frontend convenience. It's a backend protection mechanism. Persisted Queries Add Another Security Boundary GraphQL's flexibility can be reduced deliberately when an application doesn't actually need arbitrary queries. With persisted queries, the client can send an operation identifier rather than the entire query: Persisted Queries The server can then map the identifier to a previously approved query. This has several advantages. It can reduce request overhead, simplify operation-level observability, and when implemented as an allowlist dramatically narrow the set of operations that an untrusted client can execute. For public APIs, this can be a particularly powerful architectural choice. GraphQL Has an Observability Problem Here's something easy to miss. A conventional monitoring system might show: POST /graphql → 200 Great. But what actually happened? Was it: GetUser or: GetOrderHistory or: SearchProducts or an expensive administrative query? All of them may appear as: POST /graphql This means HTTP-level monitoring alone doesn't provide enough context. GraphQL needs operation-aware observability. Seeing Inside a GraphQL Request Imagine a request called: GetCustomerDashboard Underneath, it executes: GetCustomerDashboard │ ├── User resolver │ └── PostgreSQL │ ├── Orders resolver │ └── PostgreSQL │ ├── Product resolver │ └── Redis │ └── Recommendations resolver └── REST service Now suppose the request takes five seconds. You can ask a much better question: Where did those five seconds go? Perhaps the database consumed 200 milliseconds. Redis consumed 20 milliseconds. The external recommendations service consumed 4.5 seconds. Without distributed tracing, that context can be surprisingly difficult to reconstruct. GraphQL and OpenTelemetry This is where GraphQL and modern observability standards become particularly interesting. The GraphQL ecosystem has an OpenTelemetry working group focused on observability conventions and semantics for GraphQL implementations. Meanwhile, OpenTelemetry itself has continued to expand beyond conventional metrics, logs, and traces. A modern GraphQL platform can therefore look like: GraphQL and OpenTelemetry The result is much more useful than simply monitoring /graphql. You can measure latency by operation, identify slow resolvers, correlate backend dependencies, and understand which GraphQL operations are responsible for resource consumption. Observability Should Include the Schema There is another valuable signal that GraphQL provides. Because the schema describes individual fields, you can understand field usage over time. Suppose you're considering removing: User.legacyPhoneNumber Instead of guessing whether anyone still uses it, you can look at operation telemetry and determine whether clients are requesting that field. Schema governance can therefore become an observable process. You can deprecate a field, monitor its usage, communicate with remaining consumers, and eventually remove it. That's much safer than discovering after deployment that an unknown client still depends on it. Federation Changes the Architecture GraphQL becomes even more interesting when an organization has multiple teams. Imagine one team owns customers. Another owns products. Another owns orders. Another owns payments. A single GraphQL server containing every team's code eventually becomes difficult to maintain. Federation provides a different model. Federation The consumer still sees a unified graph. Internally, multiple services contribute to that graph. This allows teams to own portions of the API while providing consumers with a coherent interface. But federation also introduces new operational challenges. A slow downstream service can now affect a seemingly simple GraphQL operation. A schema change in one subgraph can affect another. Tracing becomes more important. Ownership becomes more important. Testing becomes more important. GraphQL doesn't eliminate distributed-system complexity. It moves some of that complexity behind a unified API. The Distributed Graph Needs Distributed Tracing Consider: query { customer(id: "42") { orders { products { inventory { warehouse } } } } } Behind that query could be: GraphQL Router │ ├── Customer Service │ ├── Order Service │ ├── Product Service │ └── Inventory Service If inventory suddenly becomes slow, the user experiences a slow GraphQL request. Without tracing, you might see only: GraphQL latency: 4.2 seconds With tracing, you can see: GraphQL request 4.2s ├── Customer 120ms ├── Orders 340ms ├── Products 280ms └── Inventory 3.4s Now the incident has a direction. That's the difference between monitoring and observability. GraphQL Performance Is a Full-Stack Problem When GraphQL gets slow, the problem isn't necessarily GraphQL. It could be: Client ↓ GraphQL parser ↓ Query validation ↓ Resolver ↓ DataLoader ↓ Cache ↓ Database ↓ External service Optimizing only the GraphQL server can therefore miss the real bottleneck. A mature implementation measures the entire request path. Caching can help. Batching can help. Pagination can help. Query complexity limits can help. Database indexes can help. But none of these should be introduced blindly. Measure first. Then optimize the actual bottleneck. Error Handling Matters More Than It Looks GraphQL errors are interesting because a response can contain both data and errors. For example: { "data": { "user": { "name": "Alice", "orders": null } }, "errors": [ { "message": "Unable to retrieve orders" } ] } This can be extremely useful for clients. But it creates an important security responsibility. Never expose internal errors directly. You don't want this: { "message": "PostgreSQL connection failed at 10.0.12.44:5432" } A client generally doesn't need your infrastructure details. Instead, expose a safe application-level error while retaining the technical information in internal logs and traces. The goal is: Client → Safe error Server → Detailed diagnostic context Schema Evolution Without Breaking Everyone One of GraphQL's biggest advantages is that APIs can evolve without relying on rigid URL versioning. Instead of immediately creating: /api/v2 you can introduce a new field and deprecate the old one. For example: type User { username: String @deprecated(reason: "Use displayName") displayName: String! } The old field remains available while clients migrate. The important part is observability. If you don't know who uses the deprecated field, you don't really have a safe migration strategy. A mature GraphQL platform therefore treats the schema as a living product rather than a static document. What a Production GraphQL Platform Looks Like After all of these pieces come together, the architecture starts looking less like a simple API server and more like a platform: Production GraphQL Platform Notice what's missing. There is no single magical GraphQL feature making this architecture reliable. Reliability comes from the combination of schema design, authorization, query controls, efficient resolvers, distributed tracing, infrastructure, and operational discipline. So, Should You Use GraphQL? The answer isn't simply yes. If your application has straightforward resources and predictable access patterns, REST may remain the simpler choice. But GraphQL becomes increasingly attractive when clients need flexible access to connected data, multiple consumers require different representations of the same domain, or you need a unified API over several backend systems. The important thing is not to adopt GraphQL because it is fashionable. Adopt it when its model solves a real problem. And when you do, don't stop at: "Can I run a GraphQL query?" Ask: "Can I safely run millions of them?" That's the production question. The GraphQL Mindset GraphQL has matured considerably from its early reputation as simply a flexible alternative to REST. The ecosystem now has to solve problems that appear whenever a technology becomes infrastructure: How do we secure it? How do we observe it? How do we evolve it? How do we federate it? How do we control its computational cost? How do we operate it across hundreds of services? Current GraphQL ecosystem work continues across areas including federation, incremental delivery, schema evolution, security, performance, and observability. That is why the future of GraphQL isn't really about writing clever queries. It's about building a reliable graph over increasingly distributed systems. The query is only the beginning. The real engineering happens underneath it. GraphQL gives clients something powerful: control over the shape of the data they receive. But with that power comes responsibility. A poorly designed GraphQL API can generate excessive database queries, expose sensitive information, execute expensive operations, and become difficult to debug. A well-designed one can provide a clean contract over a surprisingly complex distributed system. The difference comes down to engineering. Design the schema deliberately. Treat queries as untrusted input. Control complexity. Batch expensive operations. Enforce authorization. Paginate large collections. Instrument resolvers. Trace downstream services. Observe schema usage. Evolve the API deliberately. And most importantly, design GraphQL as a production platform, not simply as another endpoint. That's where GraphQL becomes truly powerful. Not when you write your first query. But when your system can handle the millionth one just as safely as the first. https://graphql.org/learn/introduction/

  • How to Send Email Notifications in Python Using Twilio SendGrid

    Whether you are building an alert system for your application, sending welcome emails to new users, or delivering daily reports, email automation is a core requirement for most modern applications. While Twilio is famous for SMS and voice APIs, they power email through Twilio SendGrid one of the most robust email delivery services available. Here is a complete guide to getting your Python application to send its first email using the SendGrid API. Prerequisites Before diving into the code, ensure you have: Python 3.6+ installed on your machine. A Twilio SendGrid account (the free tier allows up to 100 emails per day). Basic familiarity with python and the command line or terminal. Set Up SendGrid and Authenticate Your Domain To prevent your emails from landing in the spam folder, SendGrid requires you to verify your identity. Log into your SendGrid dashboard. Navigate to Settings > Sender Authentication. Choose Single Sender Verification (best for quick testing) or Domain Authentication (recommended for production). Fill out the required details and verify your email address via the confirmation link sent to your inbox. Note: You must use this verified email address as the "From" address in your Python script. Generate an API Key Your Python script needs permission to send emails on your behalf. In the SendGrid dashboard, go to Settings > API Keys. Click Create API Key. Give it a descriptive name (e.g., "Python App Emailer") and select Restricted Access -> Mail Send (Full Access). Copy the generated API key immediately. You will not be able to see it again once you close the window. For security, never hardcode this key in your script. Instead, set it as an environment variable in your terminal: On macOS/Linux: export SENDGRID_API_KEY="your_api_key_here" On Windows (Command Prompt): set SENDGRID_API_KEY="your_api_key_here" Install the SendGrid Python SDK SendGrid provides an official Python library that abstracts away the complexity of raw HTTP requests. Install it using pip: Bash pip install sendgrid Write the Email Script (Python) Create a new file named send_email.py and add the following code. This script builds the email object and passes it to the SendGrid client. import os from sendgrid import SendGridAPIClient from sendgrid.helpers.mail import Mail def send_notification(): # 1. Construct the email message message = Mail( from_email='your_verified_email@example.com', # Must match Step 1 to_emails='recipient@example.com', subject='System Alert: Task Completed Successfully', html_content='Success! Your background job has finished processing.' ) try: # 2. Initialize the client using the environment variable sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY')) # 3. Send the email and capture the response response = sg.send(message) # 4. Output the results print(f"Status Code: {response.status_code}") print(f"Headers: {response.headers}") print("Email dispatched successfully!") except Exception as e: print(f"Error: {e}") if __name__ == '__main__': send_notification() How the Code Works: Mail(...): This class constructs your email payload. It handles the formatting required by the API. os.environ.get(...): This safely retrieves your API key from the system environment, keeping your credentials out of version control. sg.send(message): This executes the actual POST request to SendGrid's servers. Execute and Verify Run the script from your terminal: python send_email.py If everything is configured correctly, you will see a Status Code: 202 printed in your terminal. This indicates that SendGrid has successfully accepted the message and queued it for delivery. Check the recipient inbox (and spam folder, just in case) to see your automated message. If you get an error Error: Then update the send_email.py script to below import os from sendgrid import SendGridAPIClient from sendgrid.helpers.mail import Mail import ssl import certifi # Force Python's default SSL to use certifi's updated certificate bundle ssl._create_default_https_context = lambda: ssl.create_default_context(cafile=certifi.where()) def send_notification(): # 1. Construct the email message message = Mail( from_email='your_verified_email@example.com', # Must match Step 1 to_emails='recipient@example.com', subject='System Alert: Task Completed Successfully', html_content='Success! Your background job has finished processing.' ) try: # 2. Initialize the client using the environment variable sg = SendGridAPIClient(os.environ.get('SENDGRID_API_KEY')) # 3. Send the email and capture the response response = sg.send(message) # 4. Output the results print(f"Status Code: {response.status_code}") print(f"Headers: {response.headers}") print("Email dispatched successfully!") except Exception as e: print(f"Error: {e}") if __name__ == '__main__': send_notification() Best Practices for Production: Use Dynamic Templates: Instead of writing raw HTML in your Python code, use SendGrid's Dynamic Templates to design your emails in their drag-and-drop editor and inject variables via Python. Error Handling: Implement robust logging instead of simple print() statements to track failed deliveries. Async Execution: If you are sending emails within a web framework like Django or Flask, offload the sending process to a background task (using Celery or Redis Queue) so the user doesn't have to wait for the API call to complete.

  • Introduction to Git: Basic Commands and Core Concepts

    Git is the free and open-source distributed version control system. Installation & GUIS GitHub for Windows https://windows.github.com GitHub for Mac https://mac.github.com GitHub for All Platforms https://git-scm.com Setup Configuring user information used across all local repositories Set a name that is identifiable for credit when reviewing the version history git config –global user.name “[firstname lastname]” Set an email address that will be associated with each history marker git config –global user.email “[valid-email]” Set automatic command line coloring for Git for easy reviewing git config –global color.ui auto Init Configuring user information, initializing and cloning repositories Initialize an existing directory as a Git repository git init Retrieve an entire repository from a hosted location via URL git clone [url] Stage & Snapshot Working with snapshots and the Git staging area Show modified files in the working directory, staged from your next commit git status Add a file as it looks now to your next commit (stage) git add [file] Unstage a file while retaining the changes in the working directory git reset [file] Diff of what is changed but not staged git diff Diff of what is staged but not yet committed git diff –staged Commit your staged content as a new commit snapshot git commit -m “[descriptive message]” Branch & Merge Isolating work in branches, changing context, and integrating changes List your branches.a*will appear next to the currently active branch git branch Create a new branch at the current commit git branch [branch-name] Switch to another branch and check it out in your working directory git checkout Merge the specified branch’s history into the current one git merge [branch] Show all commit in the current branch’s history git log Inspect & Compare Examining logs, diffs, and object information Show the commit history for the currently active branch git log Show the commits on branchA that are not on branchB git log branchB..branchA Show the commits that changed file, even across renames git log –follow [file] Show the diff of what is in branchA that is not in branchB git diff branchB…branchA Show any object in Git in human-readable format git show [SHA] Tracking Path Changes Versioning file removes and path changes Delete the file from the project and stage the removal for the commit git rm [file] Change an existing file path and stage the move git mv [existing-path] [new path] Show all commit logs with an indication of any paths that moved git log –stat -M Ignoring Patterns Preventing unintentional staging or committing of files Save a file with desired patterns as .gitignore with either direct string matches or wildcard globs. logs/ *.notes pattern*/ System-wide ignore pattern for all local repositories git config –global core.excludesfiles [file] Share & Update Retrieving updates from another repository and updating local repos Add a git URL as an alias git remotes add [alias] [url] Fetch down all the branches from that Git remote git fetch [alias] Merge a remote branch into your current branch to bring it up to date git merge [alias]/[branch] Transmit local branch commits to the remote repository branch git push [alias] [branch] Fetch and merge any commits from the tracking remote branch git pull Rewrite History Rewriting branches, updating commits, and clearing history Apply any commits of the current branch ahead of the specified one git rebase [branch] Clear staging area, rewrite working tree from the specified commit git reset — hard [commit] Temporary Commits Temporarily store modified, tracked files in order to change branches Save modified and staged changes git stash List stack-order of stashed file changes git stash list Write working from the top of the stash stack git stash pop Discard the changes from the top of the stash stack git stash drop

  • Why is the Cloud Changing Again?

    Remember when moving everything to "the cloud" was the ultimate tech upgrade? You finally packed up all those clunky on-site servers, sent your data off to a neat, invisible digital storage unit, and breathed a sigh of relief. But then, Artificial Intelligence exploded onto the scene, and suddenly, the old cloud rules went out the window. AI doesn't just passively store data; it actively devours it, requiring massive computing power and handling highly sensitive, proprietary information. That standard public cloud you relied on? It’s starting to look less like an open superhighway and more like a crowded city bus during rush hour. Enter the next era of enterprise tech: Cloud 3.0, Cloud Repatriation and Geopatriation. In fact, Gartner named Geopatriation a top strategic technology trend for 2026. While Cloud 3.0, Cloud Repatriation and Geopatriation might sound like intimidating buzzwords from a sci-fi novel, they actually represent a logical, necessary shift in how companies manage and protect their digital assets today. I'm breaking down exactly what these concepts mean without the heavy jargon, and why bringing data "back home" is the new frontier of the internet. What Exactly is Cloud 3.0? (The AI-Powered Cloud) If your company just finished a grueling, multi-year migration to "the cloud," hearing that there's already a version 3.0 might make you want to pull your hair out. But to understand why the infrastructure had to evolve, it helps to quickly look at how we got here. Think of the cloud's evolution in three distinct phases: Cloud 1.0 (The Storage Unit): The initial "lift and shift" era. Instead of keeping physical servers in a stuffy office closet, businesses rented server space on someone else's computer over the internet. Cloud 2.0 (The App Era): This is where most businesses live today. Applications were redesigned to live natively online. This is the era of renting software and services on demand think Google Workspace, Microsoft, Salesforce, or Netflix. Cloud 3.0 (The AI Engine): A decentralized, highly specialized network built from the ground up to handle massive, complex Artificial Intelligence workloads. The Transit Analogy: Why Cloud 2.0 Broke Think of traditional Cloud 1.0 and 2.0 like a public city bus. It is highly efficient, cost-effective, and great for getting standard daily tasks from point A to point B. You share the infrastructure with other companies, and for everyday web traffic, email, and standard software, it works perfectly. But then came modern AI. Artificial Intelligence doesn't just need a quick ride; it needs to process petabytes of heavy, complex data back and forth at lightning speed. Trying to run advanced AI training and inference on a standard public cloud is like trying to load a thousand tons of industrial steel onto that city bus. It creates massive bottlenecks, slows the entire system down, and gets incredibly expensive. Cloud 3.0 is the high-speed freight train. It is designed specifically to haul massive data loads efficiently without sharing the tracks. Rather than forcing all your data to travel to a centralized public server, Cloud 3.0 uses a mix of multi-cloud environments and "edge computing" to process data right where it lives, ensuring minimal lag. The Precursor: The Cloud Repatriation Trend Before we can understand where data is going, we have to look at a trend that has been quietly building for the last few years: Cloud Repatriation. For a long time, the assumption was that the public cloud was always cheaper. But companies eventually realized that running constant, heavy workloads 24/7 in a public cloud led to massive "cloud sticker shock." Renting is great for flexibility, but at a certain scale, owning your infrastructure is cheaper. In response, many IT leaders began repatriating - pulling specific applications and data out of global public clouds and moving them back to private, on-premises servers to regain performance stability and cut costs. Cloud repatriation proved that the public cloud wasn't a one-way street. But when AI entered the chat, this purely financial trend evolved into a legal and security mandate known as Geopatriation. Geopatriation: Bringing Your Data Back Home For years, the rallying cry in tech was "move everything to the cloud." It didn’t really matter where that cloud physically lived your company's data might have been split across a server farm in Virginia, a backup facility in Ireland, and a data center in Singapore. As long as it was accessible and cheap, global public clouds were the standard. But the rise of AI changed the rules of data privacy, giving birth to two major buzzwords you'll be hearing a lot: Data Sovereignty and Geopatriation. To understand why this is happening, let’s talk about a secret family recipe. The Family Recipe Analogy Imagine your business runs on a highly valuable, secret family recipe. For years, you kept this recipe in a massive, shared international bank vault (the global public cloud). It was secure enough, and it was convenient. However, suddenly a new machine (AI) is introduced to the vault. This machine is designed to constantly read, analyze, and learn from every piece of paper it can find to generate new ideas. Suddenly, keeping your secret recipe in a shared, international facility feels incredibly risky. What if the machine learns your recipe and uses it to help a competitor? What if the country where the vault is located changes its privacy laws? Geopatriation is the act of taking that recipe out of the international vault and putting it into a highly secure safe inside your own house. Why the sudden shift? When AI reads and learns from your data, privacy and security become paramount. This has led to a massive focus on Data Sovereignty the legal concept that digital data is subject to the laws of the country where it physically lives. Governments worldwide are passing strict new regulations requiring that citizen data remain within their own borders to protect it from foreign surveillance and unauthorized AI training. You can no longer just dump customer information into a borderless global cloud. As a result, companies are actively "geopatriating" their data. They are moving sensitive information out of centralized, global public clouds and bringing it back to local, geographically specific servers and in some cases, all the way back to private, on-site servers. The trend is moving rapidly: Gartner predicts that by 2030, over 75% of European and Middle Eastern enterprises will geopatriate their virtual workloads, up from just 5% in 2025. By bringing the data "home," businesses retain absolute control over their proprietary information, ensure compliance with local privacy laws, and guarantee that their data isn't secretly training someone else's AI. The End of the "One-Size-Fits-All" Cloud The biggest takeaway for businesses navigating this shift is that the era of a single, universal cloud strategy is over. However, this does not mean the public cloud is dying. According to recent IDC server and storage workload surveys, while nearly 80% of organizations plan to move some workloads out of the public cloud, only 8% to 9% plan a full cloud exit. Instead, industry leaders are adopting a "Three-Tier Architecture" for their data: Global Tier: Public clouds for standard web traffic and non-sensitive apps. Regional Tier: Sovereign, local clouds for regulated AI processing and financial data. Private Tier: On-premises or highly isolated servers to protect proprietary IP and secret recipes. We are moving away from a mindset where speed and cost-efficiency were the only metrics that mattered. Today, control, compliance, and risk mitigation are equally critical. Cloud 3.0, Cloud Repatriation and Geopatriation are not about abandoning the cloud or stepping backward in innovation. Instead, they represent a more mature, hybrid approach to enterprise infrastructure. You might still use a global public cloud for your everyday web traffic and standard applications, but rely on localized, sovereign servers or a private Cloud 3.0 edge network to run your proprietary AI and store your most sensitive customer data. Ultimately, the future of enterprise IT is a portfolio approach. By embracing Cloud 3.0 and bringing critical data "back home," businesses can harness the incredible power of next-generation AI without sacrificing their security, privacy, or legal control. The companies that thrive in the next decade won't be the ones that put everything in the cloud they will be the ones that put the right data in the right cloud. https://www.gartner.com/en/documents/7079998 https://www.splunk.com/en_us/blog/learn/geopatriation.html https://nhimg.org/glossary/geopatriation/ https://community.nasscom.in/communities/cloud-computing/cloud-repatriation-india-which-workloads-move-back-premise-and-how https://www.idc.com/resource-center/blog/storm-clouds-ahead-missed-expectations-in-cloud-computing/

  • How to work with DataFrame in Python?

    Python language has vast application. Data analysis and manipulation is one of many application. We can work on data using Pandas library in Python for generating data analytics and manipulating data to generate useful patterns. What is Pandas? Pandas is a Python library which is used to work with sequential and tabular data. It has functionality to manage, analyze and manipulate data in a simple and efficient way. We can think of its data structures as relatives to database tables or spreadsheets. Pandas includes NumPy library. Two primary data structures of pandas are series (1D data) and dataframe (2D data). It can work with Homogenous as well as Heterogenous data. Features of Pandas Time-series manipulation tools Works with missing data (NaN) Works with different data files (xls,db,csv,psv,hdf5,etc.) ETL tools (Extraction, Transformation and Load tools) What is DataFrame? Pandas DataFrame is a heterogenous 2D object, i.e. data are of same type within each column but it could be a different data type for each column and can be labeled with an index (implicit or explicit). In simple words, DataFrame is like table in database. The index can be implicit, starting with 0 or we can have our own index. Index can even include dates and times. Let us now work with DataFrame Creating an empty DataFrame import pandas as pd df1 = pd.DataFrame() print(df1) Creating an empty structure DataFrame import pandas as pd df1 = pd.DataFrame(columns=['Sr. no','Item','Desc']) print (df1) df2 = pd.DataFrame(columns=['Sr. no','Item','Desc'],index=range(1,10)) print (df2) Creating a DataFrame passing NumPy array import pandas as pd arr = {'Sr. no' : [1,2,3,4], 'Items' : ['A','B','C','D']} df1 = pd.DataFrame(arr) print(df1) Creating a DataFrame passing a Dictionary import pandas as pd dict1 = {1:'A',2:'B',3:'C',4:'D'} df1 = pd.DataFrame([dict1]) print(df1) Creating a DataFrame with datetime index import pandas as pd arr = {'Sr. no' : [1,2,3,4], 'Items' : ['A','B','C','D']} indx = pd.DatetimeIndex(['2021-12-30','2021-12-31','2022-01-01','2022-01-02']) df1 = pd.DataFrame(arr,index=indx) print(df1) Viewing DataFrame import pandas as pd arr = {'Sr. no' : [1,2,3,4], 'Items' : ['A','B','C','D']} df1 = pd.DataFrame(arr,index=indx) print(df1) # pd.DataFrame() also print DataFrame # get first two rows df.head(2) # get last two rows df.tail(2) # get DataFrame's index df.index # get DataFrame's columns df.columns #get DataFrame's values df.values Importing Data in Pandas Pandas DataFrame can read data from several data formats, most common includes csv, psv, xls, json, sql, hdf5, etc. We will look at few examples to import data from different data formats and customizations. import pandas as pd df1 = pd.read_csv('file1.csv',sep=' ') # to read dataframe csv file with blank space as separator df1 = pd.read_csv('file1.csv',usecols=[0,1,2,3],nrows = 100) # to read data from columns 0 to 3 df1 = pd.read_excel('file1.xls',sheet_name='User') # to read data from excel file and sheet named 'User' Once we data in DataFrame i.e. DataFrame is prepared, independent of source of data (csv,xls,db etc.) we can work with it, like it is a table in database, selecting element of our interest. import pandas as pd df1 = pd.read_csv('example.csv') df1.head() df1.shape We have loaded data from csv file and created DataFrame. Now, we will see different select operation on this DataFrame. Selecting a single column in DataFrame column1 = df1['column_name'] column1.head() Selecting multiple columns in DataFrame cols = df1['column1','column2'] cols.head() Selecting rows using indexing [] in DataFrame rows = df1[10:20] print (rows) rows = df1[10:20]['column1','column2'] print (rows) Selecting rows by lable (.loc[]) in DataFrame df2 = df1.loc[10:20] df3 = df1.loc[10:20,['column1','column2']] print(df2) print(df3) Selecting rows by position (.iloc[]) in DataFrame dfpos = df1.iloc[10:20,[3,4]] After looking at fetching required set of column we proceed to manipulation of DataFrame How to manipulate a DataFrame Transpose import pandas as pd df1 = pd.read_csv('example.csv') df2 = df1[10:20]['cols1','cols2'] print("transpose : {}".format(df2.T)) sort_values df2 = df1.sort_values(by='column_name') # sorting by single column df3 = df1.sort_values(by=['col1','col2']) # sorting by multiple column print(df2) print(df3) sort_index df2 = df1.sort_index() print(df2) Re-indexing import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(3),index=['a','b','c']) print(df1) df2 = df1.reindex([1,2,3]) print(df2) Adding a new column import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(10)) print(df1) df1['col_new'] = 'a' print(df1) Remove existing column import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(10)) print(df1) df1['col_new'] = 'a' print(df1) del df1['col_new'] # del df1[1] will work same Data at particular location by label import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(3,5),index=[1,2,3],columns=['a','b','c','d','e']) val1 = df1.at(1,'a') print(val1) df1.at(1,'a') = 0 # assign value at particular location Data at particular location by position import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(3,5),index=[1,2,3],columns=['a','b','c','d','e']) val1 = df1.iat[1,1] print(val1) df1.at[1,1] = 0 # assign value at particular location df1[df1>0] = 2 # assign data at all location based on a condition Applying a method or function in DataFrame import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(3,5),index=[1,2,3],columns=['a','b','c','d','e']) def add_five(number): return number+5 df.apply(add_five,axis=2) There are few more functions such as dropna(), fillna() etc. which is used in manipulation of data. DataFrame also have many statiscal methods like info(), describe(), value_counts(), mean(), std() etc. Information and facts are major asset in digital world. Data is fuel that is running this world. However, everything is not required by everyone. Only specific part of data is relevant for certain use cases. Thus filtering data becomes important. By filtering we can filter data as per requirement and generate pattern and knowledge to drive this digital world. import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(5,5),index=['a','b','c','d','e'],columns=['A','B','C','D','E']) print (df1) filtered_df = df1[df1['A']>0] print(filtered_df) Iterating Pandas DataFrame for item in df.iterrows(): print(item) Merge, Append and Concat operation in Pandas DataFrame DataFrame comes with feature to merge, concatenate. They are important and useful feature. import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(5,5),index=['a','b','c','d','e'],columns=['A','B','C','D','E']) print (df1) df2 = pd.DataFrame(np.random.randn(5,5),index=['a','b','c','d','e'],columns=['A','B','C','D','E']) print (df2) df3 = pd.merge(df1,df2) print(df3) append is function that allows us to add rows to existing DataFrame import pandas as pd import numpy as np df1 = pd.DataFrame(np.random.randn(5,5),index=['a','b','c','d','e'],columns=['A','B','C','D','E']) print (df1) df2 = pd.DataFrame(np.random.randn(5,5),index=['a','b','c','d','e'],columns=['A','B','C','D','E']) print (df2) df3 = df1.append(df2) print(df3) df4 = pd.concat([df1,df2]) #concat by row print (df4) Hope it helps! Happy Learning.

  • How To Get Started – Cloud Skills Boost & Arcade

    If you are new to Google Cloud and don’t know where to start this blog is for you. Google Cloud Skills Boost Google Cloud Skills Boost is the platform where you can learn a lot of cloud concepts and can choose the learning path for example if you want to learn AI you can choose the AI learning path, if you want to learn data analytics you can choose Data Analytics learning path and this is very domain focused courses but if you don’t want to go the complex route you can start with basic cloud learning path to get started. https://www.cloudskillsboost.google/paths Google Cloud Skills Boost is great but it is a bit theoretical and I like to do hands-on for any service, tool, or product. If like me you also like to get your hands dirty then I would like to introduce “The Arcade”. Welcome to Arcade Google Cloud The Arcade is the platform that offers introductory and more advanced games based on labs (https://go.cloudskillsboost.google/arcade) to help improve your cloud skills. Every month arcade provides games based on labs and courses on different cloud topics. You can complete these courses and labs to learn Google Cloud and earn points which will help to get the arcade prizes. How it works First, you have to subscribe using Email/Gmail to activate Arcade for your account. Then, join Google Cloud Skills Boost. You don’t need a new account if you’re already registered. Done. Now, play games to learn skills, earn badges, and get arcade points. I suggest starting at level 1 if you’re new to the cloud! Enroll in Skill Badges to help you complete game tasks and get more Arcade points. REDEEM points at the prize counter. Arcade Google Cloud – Trivia December 2023 Is Live Now. https://go.cloudskillsboost.google/arcade https://g.dev/chandanrajpurohit

  • ReAct in AI Explained: How Reasoning and Acting Power Modern AI Agents

    I was exploring a core concept about AI and AI Agents and frequently came across ReAct. As I kept seeing it, I thought to take a deeper dive and below is my understanding." Today’s AI models can search the web, use tools, solve multi-step problems, write code, and even interact with software environments. One of the most important concepts behind this evolution is ReAct. ReAct combines reasoning and action into a single framework, enabling AI systems to think step by step while interacting with external tools and environments. This approach is becoming a core foundation for: AI agents Autonomous systems Tool-using language models Research assistants AI copilots ReAct stands for Reason + Act It is an AI prompting and agent framework where a language model alternates between: Reasoning about a problem Taking an action Observing the result Continuing reasoning Instead of directly generating an answer, the AI follows a structured thinking process. The basic ReAct cycle looks like this: Thought → Action → Observation → Thought → Action → Final Answer This makes AI systems significantly more reliable and capable of handling complex tasks. Above should give you a basic understanding of What is ReAct now let's understand Why ReAct is important. Traditional AI models often struggle with: Hallucinations Incorrect assumptions Outdated information Complex reasoning tasks For example, if you ask a normal AI model: “Who is the CEO of the company founded by the author of The Hard Thing About Hard Things?” A basic model may guess incorrectly. But model with ReAct should give correct response. Below is the response received from ChatGPT. The author of The Hard Thing About Hard Things is Ben Horowitz. He co-founded Andreessen Horowitz (a16z) with Marc Andreessen. Andreessen Horowitz does not have a traditional CEO; it is led by its co-founders and managing partners. A ReAct-based AI system would: Identify the author of The Hard Thing About Hard Things Search for the author’s company Find and verify the CEO information Return an accurate/ correct answer This reasoning + action loop dramatically improves accuracy. How ReAct Works? ReAct combines internal reasoning with external tool usage. The workflow generally follows these stages: 1. Thought (Reasoning) The AI first thinks about the task. Example: Thought: I need to identify who wrote The Hard Thing About Hard Things. This reasoning helps the model break the problem into smaller steps. 2. Action Next, the AI performs an action using a tool. Actions may include: Web searches API calls Database queries Running code Using calculators Example: Action: Search["The Hard Thing About Hard Things author"] 3. Observation The tool returns information back to the AI. Example: Observation: Ben Horowitz wrote The Hard Thing About Hard Things. The AI then uses this information for the next reasoning step. 4. Final Answer After gathering enough information, the AI produces the final response. This iterative process allows the model to solve more advanced problems than traditional prompting methods. Real-World Applications of ReAct ReAct is widely used in modern AI systems and autonomous agents. 1. AI Agents Most modern AI agents rely on ReAct-like architectures. These agents can: Plan tasks Use tools Search for information Adapt to changing environments Examples include: Research agents Productivity assistants Autonomous copilots 2. AI Coding Assistants AI coding systems use ReAct workflows to: Analyze code Run tests Fix errors Execute commands The AI continuously reasons about problems and performs actions until the issue is solved. 3. Robotics Robots use reasoning and acting loops for: Navigation Object detection Motion planning Environment interaction A robot may: Observe an obstacle Reason about a path Move Reassess surroundings This is essentially the ReAct framework in action. 4. Customer Support Automation AI support systems use ReAct to: Search documentation Retrieve customer information Execute workflows Resolve tickets intelligently This creates smarter automated customer service experiences. Modern Large Language Models increasingly rely on ReAct-based approaches to become more autonomous and useful. Popular AI frameworks that support ReAct include: LangChain AutoGen CrewAI Semantic Kernel OpenAI Agents SDK These frameworks help AI systems: Use tools Maintain memory Coordinate tasks Execute workflows ReAct is becoming one of the foundational concepts behind next-generation AI applications. Now let's dive into some of the benefits and challenges of ReAct Benefits of ReAct Improved Accuracy By validating information through actions and observations, ReAct reduces hallucinations and incorrect responses. Better Multi-Step Reasoning ReAct handles: Logical problems Sequential tasks Dynamic environments far better than standard prompting techniques. Tool Integration The framework works naturally with: APIs Databases Search engines External software Browsers This allows AI systems to interact with the real world. Transparency ReAct exposes the reasoning process, making it easier for developers to: Debug systems Understand decisions Improve reliability Challenges of ReAct Although powerful, ReAct also has limitations. Slower Responses Because the AI performs multiple reasoning and action steps, responses may take longer. Dependency on External Tools Poor search results or unreliable APIs can reduce performance. Higher Costs More reasoning steps and tool calls increase: Compute usage API expenses Latency Error Propagation Incorrect observations can lead the AI toward wrong conclusions. ReAct is transforming how AI systems operate by combining reasoning and action into a single intelligent workflow. Instead of simply generating text, ReAct-based systems: Think step by step Use tools Observe results Adapt dynamically This approach powers many modern AI agents and autonomous systems. As the AI industry continues evolving, understanding ReAct is becoming increasingly important for developers, researchers, and businesses building next-generation AI applications. ReAct represents a major shift in artificial intelligence. The future is no longer just about generating responses - it is about creating AI systems that can reason, act, learn, and adapt continuously. For anyone interested in: AI agents Autonomous systems LLM applications Advanced prompting techniques ReAct is one of the most important concepts to understand in modern AI development.

  • Firebase Studio is Shutting Down: What You Need to Know

    With Google announcing its shutdown, many developers like me are now asking: 👉 Why is it being discontinued? 👉 What should you do next? Let’s break it all down in a simple, practical way. Firebase Studio is Shutting Down What is Firebase Studio? Firebase Studio is a cloud-based development environment (IDE) built by Google that allows developers to build full-stack applications directly in the browser. It combines: AI-powered coding assistance (via Gemini) Backend services (Firebase + Google Cloud) Frontend + mobile development tools Built-in deployment and testing tools In simple terms: It’s like VS Code + Firebase + AI copilots - all in one browser tab. Firebase Studio evolved from Project IDX and was designed to help developers prototype, build, and ship apps faster using AI and cloud infrastructure. Key Features of Firebase Studio 1. AI-Powered Development Firebase Studio integrates Gemini AI, allowing you to: Generate code using prompts Debug and refactor automatically Write tests and documentation It even supports prompt-to-app development, where you describe an app and it builds it for you. 2. Full-Stack Development in One Place You can build: Frontends (React, Angular, Vue) Backends (Node.js, Python, Go) Mobile apps (Flutter, Android) All without leaving the browser. 3. Built-in Firebase Integration It seamlessly connects with: Firestore (database) Firebase Authentication Cloud Functions Hosting This makes it ideal for rapid MVP development. 4. Cloud-Based Workspaces Each project runs in a cloud-hosted VM, meaning: No local setup required Works on any device Easy collaboration 5. Templates + Rapid Prototyping You can: Start from templates Import GitHub projects Generate apps from Figma designs Why Firebase Studio Was Popular Firebase Studio stood out because it: Removed local setup complexity Enabled fast prototyping with AI Integrated frontend + backend seamlessly For startups and indie developers, it was a “build fast, ship faster” tool. Firebase Studio Sunset: What’s Happening? Google has officially announced that Firebase Studio is being discontinued. Important Dates March 19, 2026 → Sunset announced June 22, 2026 → New workspace creation disabled March 22, 2027 → Full shutdown Why is Firebase Studio Shutting Down? Google is consolidating its AI development tools. Instead of maintaining Firebase Studio, it is shifting focus to: Google AI Studio (for AI prototyping) Google Antigravity (for advanced development workflows) The goal is to simplify the ecosystem and integrate AI features more deeply into core platforms. Source: - (https://timesofindia.indiatimes.com/technology/tech-news/google-announces-it-is-shutting-down-firebase-studio-less-than-a-year-after-its-launch-read-the-message-to-users/articleshow/129694408.cms) Important: Firebase itself is NOT Shutting Down This is where many developers get confused. Only Firebase Studio (the IDE) is shutting down Firebase services (Firestore, Auth, Hosting) will continue normally So your backend infrastructure is safe. How to Migrate from Firebase Studio Based on the official migration guidance, here’s what you should do: 1. Export Your Code Download your project or sync with GitHub Ensure all changes are committed 2. Choose a New Development Environment Recommended options: Google AI Studio (AI-first workflows) Google Antigravity (advanced dev environments) VS Code + Firebase CLI (manual setup) 3. Reconnect Firebase Services Your Firebase project remains intact, so you can: Reuse Firestore databases Keep authentication setups Continue using hosting 4. Rebuild Your Workflow Since Firebase Studio handled everything in one place, you may need to: Set up local environments Configure CI/CD pipelines Integrate AI tools separately What Developers Should Do Now If you’re currently using Firebase Studio: ✔ Start migration early ✔ Back up all projects ✔ Test alternatives before deadlines ✔ Avoid creating new dependencies on the platform Know more about the Firebase Studio sunet and migration guide at https://firebase.google.com/docs/studio/migrating-project Firebase Studio was an ambitious attempt to redefine how apps are built - combining AI + cloud + full-stack development into a single experience. Even though it’s being sunset, its ideas are not going away.They’re simply being absorbed into more powerful tools. The future of development is still AI-assisted, cloud-native, and faster than ever - just not under the Firebase Studio name.

bottom of page