We believe empowering engineers drives innovation.
We believe empowering engineers drives innovation.
At Rearc Data, we build a wide variety of data pipelines from hard-to-process sources. These include PDF reports, web dashboards, and free text. Often, there is some pattern or visual guide that enables us to extract data cleanly. Other times, the data we want is entirely unstructured in that context, and there’s no “right” way to extract it. To illustrate this problem, let’s use a real-world use case. One of our Health and Life Sciences (HLS) datasets pulls weekly influenza data from a PDF report published by a Spanish agency.
What should be done when one’s Terraform turns into technical debt? Table of Contents The Necessity of Infrastructure as Code Platform Engineering Case Study: The Successful Platform Team Debt Collector on Line One Requirements for a Solution Existing Tools Fall Short Airflow: More Than an ETL Tool Bronco: Managing Terraform at Scale with Airflow Terraform Upgrades as Directed Acyclic Graphs Concurrency, Monitoring and Retries The X Factor Summary The Necessity of Infrastructure as Code Cloud computing has fundamentally transformed how computer software is run.
Cloud Configuration ELT with CloudQuery This article discusses the purpose of CloudQuery, some of its use cases and a simple Quick Start that can be followed along to get results quickly. Table of Contents Table of Contents What is CloudQuery Use Cases for CloudQuery CloudQuery Architecture Brief Quick Start Quick Start Single Account Quick Start Multi Account Prerequisites Setup Quick Start Project Setup Database Setup CloudQuery Configuration Execute Sync Review Data Summary Challenges Outlook Suggestions For Further Exploration What is CloudQuery CloudQuery is a data integration framework that is open-source and highly performant.
Onboarding a new application to any customer identity as a service platform, such as Auth0 or AWS Cognito is simple and straightforward; however, existing customer identities in a brownfield development scenario (for example an existing application with active users where we want or need to replace the existing identity platform) can pose a challenge - how do you migrate these customers and applications to a new identity platform with minimal impact to the user experience?
What is Cloud Governance? Cloud governance includes the policies, procedures, and controls to manage cloud resources. They ensure that organizations use resources in line with their goals and objectives. Why Do You Need Cloud Governance Cloud governance has many benefits, including: Cost Optimization: Cloud computing’s scalability makes increasing or decreasing resources easy. The ease of scalability can result in wasteful expenditures if you do not actively remove unused resources. Cloud governance monitors usage and identifies savings opportunities, helping organizations optimize their budget.
Introduction Tools to improve the development experience have existed for decades. From syntax checking to code formatting, refactoring to auto-completion, developers working within an IDE frequently make use of extensions to provide additional functionality which will improve their productivity. These development extensions have recently been enhanced by the possibilities driven by AI, to provide better recommendations to developers. As infrastructure engineers, let’s explore some of those new tools and how they interact with Infrastructure-as-Code (IaC) tasks.
The beauty between noise and silence We’ve become accustomed to controlling the notifications of our lives. We tap unsubscribe on emails that offer it, we become frustrated with apps demanding too much attention, and we escape our typical work environment to that sweet single-person meeting room to get some uninterrupted work done. We crave peace. By contrast however, it’s the very nature of joining society that we must pull up a chair and turn up the volume high enough to catch the next wave.
Introduction Have you ever written terraform code and found that your code paths covered edge cases that were only in production? Or perhaps, written a reusable terraform module that you needed some way to test and prove worked as expected? When you write software, it’s very often expected to write tests to prove the software is works as intended and is relatively bug free. With Infrastructure as Code, testing is often an afterthought, if ever even considered, which can make testing such things difficult.
Introduction & Background Hi, I’m Mueez Khan. I’m currently an undergraduate Computer Science student at Rutgers University, and I was a remote summer intern at Rearc from June to September, 2022. In this article we’ll go over various projects and achievements during my internship at Rearc. About Rearc Rearc is a boutique cloud software & services firm with engineers that have years of experience shaping the cloud journey of large scale enterprises.