We believe empowering engineers drives innovation.
We believe empowering engineers drives innovation.
At Rearc Data, we build a wide variety of data pipelines from hard-to-process sources. These include PDF reports, web dashboards, and free text. Often, there is some pattern or visual guide that enables us to extract data cleanly. Other times, the data we want is entirely unstructured in that context, and there’s no “right” way to extract it. To illustrate this problem, let’s use a real-world use case. One of our Health and Life Sciences (HLS) datasets pulls weekly influenza data from a PDF report published by a Spanish agency.
Large language models (LLMs) have introduced a whole new world of opportunities. Suddenly, the interface between machines and human communication has broken down, and ideas that were recently untenable have become almost trivially simple. It’s an exciting time, for sure. Real success, however, depends on wrapping this new technology with software that makes it truly useful. One pattern that has arisen to make that happen is Retrieval Augmented Generation (RAG).
A picture is worth a thousand words - Fred R. Barnard In the rapidly changing world of LLMs, chatbots, and RAGs (Retrieval-Augmented Generation), most of the focus has been on plain text data. However, a wealth of information is available only in image form. What does something look like? How do elements of a diagram fit together? How is the food pyramid laid out? How do two algorithm outputs vary in visual quality?
Introduction Integrating Azure Active Directory (AD), now called Microsoft Entra ID Connect, and AWS IAM Identity Center to enable single sign-on (SSO) can be a long and unclear process. One of the necessary steps in this process is configuring Azure AD to send information to AWS IAM Identity Center that it requires, when the two exchange information (referred to as provisioning in Azure AD). If this step is not performed correctly, provisioning will fail.
What is Cloud Governance? Cloud governance includes the policies, procedures, and controls to manage cloud resources. They ensure that organizations use resources in line with their goals and objectives. Why Do You Need Cloud Governance Cloud governance has many benefits, including: Cost Optimization: Cloud computing’s scalability makes increasing or decreasing resources easy. The ease of scalability can result in wasteful expenditures if you do not actively remove unused resources. Cloud governance monitors usage and identifies savings opportunities, helping organizations optimize their budget.
Introduction & Background Hi, I’m Mueez Khan. I’m currently an undergraduate Computer Science student at Rutgers University, and I was a remote summer intern at Rearc from June to September, 2022. In this article we’ll go over various projects and achievements during my internship at Rearc. About Rearc Rearc is a boutique cloud software & services firm with engineers that have years of experience shaping the cloud journey of large scale enterprises.
Prompted by an uptick in high profile outages, one product engineering team at a global investment bank began to explore options for increasing the resiliency of their services. With a certainty of rapid growth as the team prepares to onboard product data and consumers from several disparate teams across the firm, the team began to look towards a migration to Amazon Web Services for their unified platform. Comprised of a small handful of specialized software engineers, experts in their domains but unfamiliar with cloud infrastructure, the product team engaged with Rearc to solve their needs.
Terraform is no doubt a popular go-to for developers looking to manage their infrastructure as code. It’s powered by HCL, a declarative configuration language from HashiCorp, that in many cases offers a powerful way to deploy infrastructure where the author can just focus on the “what” (desired state) and let the tool figure out the “how” to reconcile changes needed in the live environment. Sometimes though, operations personnel require more precise control over how changes get applied in order to minimize impact to live systems.
Flow logs are the native network logging layer for AWS. These logs can be setup specifically for logging IP traffic on subnets, network interfaces, or VPCs. VPC flow logs in particular contain a vast amount of IP traffic information and data points for our resources that can be leveraged for: Monitoring boundaries for networks and AWS accounts Detecting anomolous network activity Catching unintentional cross-region data transfers early (to avoid unnecessary costs) Identifying system optimizations based on AZ distribution Performing various network traffic flow optimizations In this blog post, we’ll be learning how to:
Introduction Migrations are one of the primary focuses for a company when they’ve decided to move to the cloud. Not only is it a shift in who is hosting your data and applications, but is also a phenomenal opportunity to modernize, optimize, and fundamentally improve the technology stacks that support business processes. This blog post details some of the benefits of moving to the cloud along with some tips to make the move as cost-effective as possible.
Performing any data migration at scale is a delicate calculation of technical complexities and cost. AWS has an extensive suite of database services to assist with these efforts in their Relational Database Service (RDS) offerings. In this post, we discuss the Aurora Global Database product and how it was used to migrate applications for a large financial media customer. We discuss the pros and cons of the product as it relates to other RDS offerings and why this was the best solution for our use case.
