Data Discovery + Access Control + Encryption

If you are planning to migrate analytical workloads from on-prem to the public cloud then you must read this article. What I have encountered in the last 9 months is that organizations moving their data to the cloud not only struggle with ETL but also with an end to end data protection process which is comprised of Auto Data Discovery, Access Control, and Encryption.

What does Auto Data Discovery mean?

The goal of ETL jobs is to land data at the storage layer which can be ADLS, S3, GCS, or any of the…

Azure Kubernetes Service (AKS) offers serverless Kubernetes, an integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance. Unite your development and operations teams on a single platform to rapidly build, deliver, and scale applications with confidence. Source

Privacera provides an enterprise solution to provide centralized data governance and access management across all of enterprise data services.

This article is divided into 3 different parts.

Part 1 — Prerequisites

Part 2 — Setting up AKS, K8 and Helm

Part 3 — Privacera installation

Part 1

Prerequisites:

Azure Client

az login → configure azure cli with your account

…

Try this :)

[root@xx ~]# crontab -l
* * * * * /root/hostname.sh

[root@xx ~]# cat /root/hostname.sh
#!/bin/bash
hostname newhostname

I did try the following but no luck

cat /usr/share/dracut/modules.d/99base/parse-hostname.sh
type hostname >/dev/null 2>&1 || \
hostname() {
if [ -n “$1” ]; then
printf — “%s” “$1” > /proc/sys/kernel/hostname
else
cat /proc/sys/kernel/hostname
fi
}

if hname=$(getarg hostname=); then
hostname “$hname”
fi

cat /proc/sys/kernel/hostname

cat > /proc/sys/kernel/hostname

new hostname

control+D

hostname

hostname -f

Azure Synapse is a scalable analytics service that brings together enterprise data warehousing and Big Data analytics capabilities. It gives users the freedom to query data on their terms, using either serverless or provisioned resources at scale. Azure Synapse brings these two operating models together with a unified experience to ingest, prepare, manage, and serve data for business intelligence (BI) and machine learning (ML)use cases. Source

This article provides an overview of Privacera’s “Policy Sync” module which delivers fine-grained access control for Azure Synpase. …

Power BI is a collection of software services, apps, and connectors that work together to turn your unrelated sources of data into coherent, visually immersive, and interactive insights. Your data may be an Excel spreadsheet or a collection of cloud-based and on-premises hybrid data warehouses. Power BI lets you easily connect to your data sources, visualize and discover what’s important, and share that with anyone or everyone you want. Source

This article explains the integration of PowerBI with Databricks and how fine-grained access control take effect which is having table, column, and row-level access controls.

Let’s connect to Spark data…

Qubole is a cloud-native data analytics platform that supports a number of enterprise-grade data processing engines such as Apache Spark, Presto, Hive, Quantum, Airflow, and more. It is used by companies like Expedia, Under Armour and Adobe.

As its popularity grows, more and more users from different departments with different roles across the enterprise are accessing data stored in Qubole. This increases the need for robust data access governance capabilities to comply with regulations like GDPR and CCPA.

Privacera, based on Apache Ranger, enables IT and data platform teams to automatically discover and classify sensitive data, define and enforce access…

Launch EC2