In this blog I will go over how to use S3 storage on a Pure Storage Flashblade with Trino the fast distributed SQL query engine for big data. I deploy Trino using the hive chart and provide a values.yaml file with the following configuration: additionalCatalogs: lakehouse: |…

In this blog I will cover how to setup Hive metastore on K8S and then leverage external S3 datasets. Installation In order to deploy Hive components on the Kubernetes cluster I first add the required helm chart repo: $ helm repo add bigdata-gradiant https://gradiant.github.io/bigdata-charts/ $ helm repo update I can then…

In this blog I will go over how you can use fast NFS and S3 from Pure Storage to power your Dremio K8S deployments. Dremio distributed storage First I change the distStorage section in the values.yaml …

In this blog I will show a simple implementation of Airbyte on kubernetes with S3 integration on PureStorage Flashblade. From our Kubernetes server with Helm installed I first add the required helm repo for Airbyte: helm repo add airbyte https://airbytehq.github.io/helm-charts helm repo update Then I deploy with, if required, a…

Intro With a steadily increasing number of businesses leveraging object based services, the amount of data stored in S3 buckets has been exploding. These repositories often contain millions or even billions of items, making it an uphill task to quickly list and manage S3 objects, without some prefix rules implemented. …

Posting this quick code snippet for those who may be looking to export metrics with a timestamp for scraping by prometheus. This can be useful when parsing metrics that are provided with a certain “lag”, that we still want visible within prometheus -> grafana. const ( timelayout = "2006-01-02…

The following article will cover various NFS mount options and the results obtained against two NFS servers, one a Virtual Machine and the other an external NFS storage array. Prior to diving into the results the test environment used was as follows: A client VM on esxi hostA with a…

I found some time a few months back to look into VMware vCloud Directors Object Storage Extension VMware vCloud Director OSE acts as a frontend for all S3 management and API calls. By this I mean that a tenant user uploading data to their S3 bucket will be communicating with…

In this blog I will cover how to access file or object storage from an MLrun job. The entire process was done using the Iguazio and Pure Storage MLOps platform as per this post. As per the documentation MLRun is an open-source MLOps framework that offers an integrative approach to…