Data Quality CSV

I have been working on automated data ingestion activity on snowflake. The python code uses Snowflake INFER_SCHEMA with Evolution Changes to load the data. data-load with schema evolution The system is set up to automatically figure out the file metadata schema from a group of staged data files that contain semi-structured data, and then get the column definitions. The GENERATE_COLUMN_DESCRIPTION function uses the result of the INFER_SCHEMA function to make it easier to create new tables, external tables, or views (using the right CREATE command) based on the column definitions of the staged files....

April 26, 2024 · 2 min

Basic Notes on Containerization

Few Tips : Whenever config-map is updated, you need to manually restart the services to update the changes Ephemeral Volumes exist for the lifespan of a Pod. Persistent Volumes preserve data beyond the lifespan of a Pod. Volume Types emptyDir - Empty directory in Pod with read/write access. Only persisted for the lifespan of a Pod. A good choice for cache implementations or data exchange between containers of a Pod. hostPath: File or directory from the host node’s filesystem....

March 13, 2024 · 3 min

Github Foundation

Recently, GitHub has introduced certifications to demonstrate expertise in GitHub technologies and workflows. Acquiring a GitHub Certification may provide a competitive advantage in the job market. This post summarizes few basic of Github Foundation What is Github? Github is VERSION CONTROL SERVICE that initially offered to hosting managed remote git repositories ad has expanded their service offerings around the hosted codebase Git Repository Hosting Project Management Tools (projects and projects classic) Issue tracking Pull requests and Code review Github Pages (for hosting static sites) Github Actions Github Copilot Github Codespaces Github Marketplace Github Gists Github Discussions Colloboration Features (Orgs and Teams) API access and Dev tools (Github Desktop and CLI) Security Features - (code scanning, secret scanning etc) Educatonal resources and course automation ⭐️ Fun fact - Github is originally built using Ruby on Rails...

March 8, 2024 · 9 min

K8s - CKAD Exam Tips and Notes

The CKAD exam is about more than just your knowledge of Kubernetes. It also tests your ability to complete tasks quickly. Speed ​​and efficiency are key to passing this test. This exam truly tests the pratical working knowledge and your proficiency. Please read the important instructions about device you use for taking exam, protocor requirements, exam details etc Few Tips : Please run PSI Online Proctoring System Check before the exam to ensure no surprises on exam day....

February 14, 2024 · 7 min

Geospatial Tech Radar - 2024

Recently came across the geospatial tech radar article published by Element84. This was widely recieved with in gis community and many conversations/discussions happened in social media (twitter/linked-in). The radar aims to highlight technologies that are proven and mature (Adopt), technologies that are ready for use but not fully proven (Trial), technologies worth looking at closely but not necessarily for trial (Assess),and technologies that haven’t happened yet but have forthcoming developments (Watch)....

February 13, 2024 · 2 min