Parsing Dynamic JSON Data in Power Query: A Quick Coding Guide

Learn how to efficiently convert any JSON data into a fully expanded table format using Power Query, reducing the need for custom code and streamlining your data processing. Introduction: Understanding JSON Data and Power Query  JSON (JavaScript Object Notation) is a lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate. It’s widely used in web services and APIs to exchange data between clients and servers. JSON’s structure typically includes nested arrays and objects, making it flexible for various types of data but also challenging to work with when it comes to data transformation and analysis.  Power Query is a data connection technology that enables you to discover, connect, combine, and refine data across a wide variety of sources. It is integrated into Microsoft products like Excel and Power BI, providing users with a powerful tool to transform and prepare data for analysis. Power Query’s ability to automate data transformation tasks and handle large datasets makes it an essential tool for business intelligence professionals.  The Challenge with JSON Data  JSON (JavaScript Object Notation) is a common data format, especially in business contexts where data is exchanged between systems. While JSON’s nested structure is useful for organizing data, it can be challenging to work with tools like Power BI, where a flat table format is often required for analysis. Traditionally, handling varied JSON structures involves writing custom code for each case, which can be time-consuming and difficult to maintain. To address this, we developed a Power Query solution that dynamically converts any JSON data into a fully expanded table format. This blog explains how the solution works, provides the code, and highlights the key logic behind it.  The Problem: Handling Varied JSON Structures  JSON files often contain nested records and lists, making it difficult to transform them into a usable table format. Manually writing different code for each variation of JSON data can be tedious and error-prone, especially when dealing with data from multiple sources or systems.  Our goal was to create a solution that could handle any JSON structure, automatically expanding it into a flat table without the need for constant code adjustments.  The Solution: Dynamic JSON Parsing with Power Query  We created a two-part Power Query script—JsonParser and ColumnExpander—that dynamically parses and expands JSON data into a table. Here’s a brief overview of how each script works:  JsonParser: Loading and Parsing JSON Data  The JsonParser script is responsible for loading the JSON data from a SharePoint site, parsing it, and preparing it for expansion.  let JsonParser = (FileNameWithoutExtension as text) => let JsonFileName = FileNameWithoutExtension & “.json”, SharePointFilesList = SharePoint.Files(“https://YourSharepointSiteURL/”, [ApiVersion = 15]), FileContent = SharePointFilesList{[Name=JsonFileName, #”Folder Path”=”https://YourSharepointFolderURL”]}[Content], JsonParsedData = Json.Document(FileContent, 1252), DataAsTable = if JsonParsedData is record then Record.ToTable(JsonParsedData) else if JsonParsedData is list then Table.FromList(JsonParsedData, Splitter.SplitByNothing(), null, null, ExtraValues.Error) else null, ExpandedData = if DataAsTable <> null then ColumnExpander(DataAsTable, 0) else null in ExpandedData in JsonParser Key Points: ColumnExpander: Expanding All Columns  The ColumnExpander script is designed to recursively expand any nested lists or records in the table, ensuring that the JSON data is fully flattened into a table format.  let   ColumnExpander = (TableToExpand as table, ColumnToExpand as number) =>     let       CurrentColumnIndex = ColumnToExpand,       CurrentColumnName = Table.ColumnNames(TableToExpand){CurrentColumnIndex},       CurrentColumnContents = Table.Column(TableToExpand, CurrentColumnName),       HasRecords = List.Contains(List.Transform(CurrentColumnContents, each _ is record), true),       HasLists = List.Contains(List.Transform(CurrentColumnContents, each _ is list), true),       CanExpandColumn = HasRecords or HasLists,       IntermediateTable =         if CanExpandColumn then           if HasRecords then             let               RecordFieldsToExpand = List.Distinct(                 List.Combine(                   List.Transform(                     CurrentColumnContents,                     each if _ is record then Record.FieldNames(_) else {}                   )                 )               ),               NewExpandedColumnNames = List.Transform(                 RecordFieldsToExpand,                 each CurrentColumnName & “.” & _               )             in               Table.ExpandRecordColumn(                 TableToExpand,                 CurrentColumnName,                 RecordFieldsToExpand,                 NewExpandedColumnNames               )           else if HasLists then             Table.ExpandListColumn(TableToExpand, CurrentColumnName)           else             null         else           TableToExpand,       FullyExpandedTable =         if CanExpandColumn then           ColumnExpander(IntermediateTable, CurrentColumnIndex)         else if CurrentColumnIndex = (Table.ColumnCount(IntermediateTable) – 1) then           IntermediateTable         else           ColumnExpander(IntermediateTable, CurrentColumnIndex + 1)     in       FullyExpandedTable in   ColumnExpander Key Points: Why This Solution Matters  This Power Query solution simplifies the process of working with JSON data. Instead of writing new code for each variation in your data, you can use these scripts to automatically flatten any JSON structure into a table. This saves time, reduces errors, and streamlines your data processing workflow.  Conclusion  Transforming JSON data into a usable format doesn’t have to be complicated. With the JsonParser and ColumnExpander scripts, you can automate the process of expanding JSON data into a table, allowing you to focus on what really matters—analyzing and using your data.  If your business is dealing with complex JSON data and needs a reliable, scalable solution, contact us today to learn how we can help you streamline your data processing workflows.  FAQs Q1: What exactly is this solution designed to do?  The solution suggested above is designed to dynamically transform JSON data of varying structures into a flat, table format using Power Query. It automates the expansion of nested records and lists within the JSON, eliminating the need for manual adjustments or custom coding for each new JSON structure.  Q2: How does this solution differ from traditional JSON processing methods  Traditional JSON processing often involves writing specific code for each unique JSON structure, which can be time-consuming and prone to errors. This solution, however, is flexible and dynamic, allowing it to handle any JSON structure automatically. It simplifies the process by recursively expanding all nested elements, making the data ready for analysis without the need for manual intervention.  Q3: Can this solution handle any type of JSON data?  Yes, it can dynamically expand any JSON data, whether it contains nested records, lists, or a mix of both.  Q4: Is this solution scalable for large datasets?  Absolutely. Power Query is optimized for handling large datasets efficiently, although performance may vary depending on the complexity of the JSON.  Q5: How does this compare to other JSON parsing methods in Power BI?  Most methods require manual adjustments for different JSON structures. This solution automates that process, making it more efficient and reducing the risk of errors.  Q6: Can these scripts be customized for

Read More

Decoding Q*: A Comprehensive Exploration for Both Technical and Non-Technical Minds

Today, we’re on our journey to decoding Q* (Q-Star), a name that sparks interest and curiosity. But what’s behind this mysterious combination of letters and symbols? What This Blog Will Cover Dive into the origin of Q* (Q-Star) and its intriguing blend of Q-Learning and A* algorithms. Ready for an adventure at the crossroads of machine learning and pathfinding? Let’s embark on this exciting exploration together! Let’s Go… Fusion of Q-Learning and A-Star The name Q* (Q-Star) likely draws inspiration from two powerful algorithms: Q-Learning and A* (A-Star). Let’s break it down for both tech enthusiasts and novices alike! Q-Learning Unveiled Q-Learning is like teaching a pet new tricks but for computers! Here’s the gist: Environment and Agent: Picture a video game or maze as the environment, and the AI as your in-game character. In simpler terms, it’s like mastering a video game by learning and adapting over time! A* (A-Star) Magic A* is a wizard in finding the shortest path between two points. Imagine it as your GPS for optimal routes: Now that we’ve got the basics, let’s get a bit speculative. How do these algorithms fit into the realm of large language models and AI? Decoding Q* (Q-Star) for Language Models Current large language models (LLMs) have limitations, especially in creative problem-solving and long-term strategy. Here’s where Q* comes into play: In the language model landscape, Q* could revolutionize how AI learns from interactions, improving responses, and adapting to new information and feedback over time. Excitingly, OpenAI’s breakthrough in Q-learning might just usher in a new era for large language models, overcoming current limitations and paving the way for unprecedented advancements in AI. It’s a thrilling journey where machine learning meets pathfinding, potentially transforming how AI systems tackle complex tasks. And who knows, Q* might be the missing piece for the next big leap in AI evolution! What are your thoughts on Q* and its potential?

Read More

Exploring the Power of Quantum Transfer Learning

In today’s world of deep learning, transfer learning is a well-versed technique to train artificial neural networks. One of the major reasons is that it is good at solving complex problems and another is the need for less training data. The recent developments in Quantum computing opens some new areas such as Quantum machine learning which leverages the power of quantum computing with traditional neural network to enhance its accuracy and help in exploring new patterns in the data. In this article, we will deeply explore the power of Quantum transfer learning and where it can be utilized. We will also show how to implement a variational quantum circuit to use quantum principles with transfer learning. Before going into depth, we need to get an understanding of what is transfer learning and its related terminologies  In quantum transfer learning, the core concept involves transferring knowledge or insights acquired from one quantum task to another, even if the tasks are distinct. This knowledge transfer serves to enhance the performance or expedite the learning process for the target quantum task. A significant departure from classical transfer learning lies in the replacement of LSTM cells with variational quantum circuits, a unique type of quantum circuit. Basic Terminologies For better understanding, we should be familiar with the basics of neural networks, transfer learning, and some basics of Quantum computing. Transfer learning Transfer learning (TL) is a technique in machine learning (ML) in which knowledge learned from a task is re-used to boost performance on a related task. For example, for image classification, knowledge gained while learning to recognize cars could be applied when trying to recognize trucks. This topic is associated with the psychological literature on transfer of learning, although practical ties between the two fields are limited. Reusing/transferring information from previously learned tasks to new tasks has the potential to improve learning efficiency significantly. Quantum Computing and related terminologies Quantum computing The term ‘Quantum’ comes from Quantum Mechanics, which is the study of the physical properties of the nature of electrons and photons in physics. It is a framework to describe and understand the complexities of nature. Quantum computing is the process of using quantum mechanics to solve highly complicated problems. We use classic computing to solve problems that are difficult for humans to solve. Now, we use quantum computing to solve problems that classic computing cannot solve. Quantum computing works on a huge volume of complex data in quick time. Quantum Superposition Superposition is when the quantum system is present in more than one state at the same time. It’s an inherent ability of the quantum system. We can consider the time machine as an example to explain superposition. The person in the time machine is present in more than one place at the same time. Similarly, when a particle is present in multiple states at once, it is called superposition. Quantum Entanglement Entanglement is the correlation between the quantum particles. The particles are connected in a way that even if they were present at the opposite ends of the world, they’ll still be in sync and ‘dance’ simultaneously. The distance between the particles doesn’t matter as the entanglement between them is very strong. Einstein had described this phenomenon as ‘spooky action at a distance. Qubits A quantum bit is a measure of data storage unit in quantum computers. The quantum bit is a subatomic particle that can be made of electrons or photons. Every quantum bit or Qbit adheres to the principles of superposition and entanglement. This makes things hard for scientists to generate Qbits and manage them. That’s because Qbits can show multiple combinations of zeros and ones (0 & 1) at the same time (superposition). Quantum Transfer Learning Quantum transfer learning is a concept that draws inspiration from classical machine learning and transfer learning but is applied in the context of quantum computing. Quantum computing leverages the principles of quantum mechanics to perform certain computational tasks more efficiently than classical computers. In the context of quantum transfer learning, the idea is to transfer knowledge or information learned from one quantum task to another, potentially different quantum task. This transfer of knowledge can help in improving the performance or speeding up the learning process for the target quantum task. One of the major differences between classical transfer learning and Quantum transfer learning is here we are replacing LSTM cells with variational quantum circuits which are a kind of quantum circuit. Here is a basic workflow of Quantum transfer learning: The architecture of Quantum transfer learning is given below: Quantum transfer learning has the potential to address challenges in quantum machine learning, such as the limited availability of quantum training data and the computational cost of training quantum models from scratch. By transferring knowledge from related tasks, it may be possible to achieve better results and accelerate the development of quantum algorithms. It’s important to note that quantum transfer learning is an emerging field, and its practical applications and effectiveness are still being explored. As quantum computing technology advances, we can expect more research and developments in this area. Use cases of Quantum transfer learning Quantum transfer learning, although an emerging concept, holds promise for a variety of use cases in quantum computing and machine learning. Some potential use cases include: Quantum Chemistry Transfer learning can be applied to quantum chemistry problems. For instance, knowledge gained from simulating the electronic structure of one molecule can be transferred to another molecule, accelerating the calculation of quantum properties. Quantum Optimization Quantum transfer learning can be used to enhance quantum optimization tasks. Knowledge from solving one optimization problem can be transferred to solve similar problems more efficiently. Quantum Image Processing In quantum image processing, knowledge learned from one type of image analysis task can be transferred to others, such as object recognition, image denoising, or image segmentation. Quantum Natural Language Processing (QNLP) Transfer learning can be valuable in QNLP tasks. Information gained from one quantum NLP task (e.g., sentiment analysis) can be transferred

Read More

How to Make Stock Price Predictions Using Reinforcement Learning?

Analyzing the stock market using artificial intelligence has been a work in progress recently. Here, we’ll discuss how you can develop an AI stock prediction model using reinforcement learning.  Analyzing the behavior of the stock market has been a subject of interest and challenge in the AI industry. Data scientists, market analysts, and financial experts have been curious to determine whether it is possible to overcome these challenges. The biggest concern is the need for an extra large dataset to build a predictive system based on supervised learning algorithms.  Furthermore, even the most advanced technologies seem to be inadequate to accurately predict the changing prices in the stock market. Yet, accurate AI stock price prediction could be possible without relying on large datasets.  In this blog, we’ll try to identify the challenges of stock market prediction and understand if we can use reinforcement learning for stock prediction and data analysis in Python, that too, using limited or no data to train the algorithm.  Before we proceed to read more about stock price prediction using machine learning, let’s understand more about the data analysis methods used to process stock market data.  Types of Data Analysis Techniques Used on Share Market Data  The stock market data is analyzed in different techniques. These are categorized as – Time Series Analysis and Statistical Data Analysis. 1. Time Series Analysis  A time series is defined as a sequence of data points that appear/ occur in successive order in a given period. It is the opposite of cross-sectional data, where the events that occur at a specific time point are captured.  The time series analysis tracks the movement of the chosen data points over the specified period. The data points are usually the price of the stock/ share/ security. The prices are collected at regular intervals to analyze the patterns.  There are various techniques to perform the time series analysis on stock market data. Let’s check them out in brief.  a. Moving Averages:  The moving average of a stock is calculated to smooth the price data and constantly update the average price. In finance, the MA (moving average) is considered a stock indicator and is used in technical analysis. The short-term price fluctuations are mitigated in this process. The MA is further divided into the following:  i. Simple Moving Average (SMA)  SMA is calculated using the arithmetic mean for a given set of values over a specific period. Here, the set of values is the stock prices. These are then added and divided by the number of prices in the set.  Formula: A1+ A2+ A3+… Ann Here, A= average in the period; nn= number of periods; SMA= n   ii. Exponential Moving Average (EMA)  The EMA gives more importance to recent prices to make the average price more relevant based on the new information. The SMA is calculated first to use in the EMA formula.  The smoothing factor is calculated next to determine the weighting of EMA- 2/(selected period+1). Formula: EMAt= [Vt×(1+ds)]+EMAy×[1−(1+ds)] Here, EMAt= today’s EMA; Vt= today’s value; EMAy= yesterday’s EMA; ds= smoothing (number of days) Some other types of moving averages are:  b. ARIMA:  It is another approach to time series forecasting. ARIMA and exponential smoothing are widely used methods as they offer a complementary approach to the problem. ARIMA describes the auto-correlations in data, while exponential smoothing relies on seasonality in data and trend description.  c. Box Jenkins Model:  This model can analyze different types of time series data for forecasting. It is a mathematical model that uses inputs from specified time series to forecast data ranges. The Box Jenkins model determines the outcomes based on the differences between data points. It identifies trends for forecasting stock prices using autoregression, moving averages, and seasonal differences.  d. Rescaled Range Analysis:  It is a statistical technique developed to assess the magnitude and nature of data variability over a certain period. The rescaled range analysis method is used to identify and evaluate persistence, randomness, and mean reversion based on the time series data from the stock markets. This insight is used to make proper investment strategies.  2. Statistical Data Analysis  It is the common value that occurs in the dataset.  It is the middle number in the dataset. For example, in 4, 6, 7, 9, and 11, the median is 7.  It is the average value of the dataset.  It is also called standard normal distribution or Gaussian distribution model. It is charted along the horizontal axis, representing the total value spectrum in the dataset. The values of half the dataset will be higher than the mean, while the other half will be longer than the mean. And the other data points will be around the mean, with a few lying on extreme/ tail ends on both sides.  It measures the asymmetry/ symmetry of the price/ data point distribution. The skewness will be zero in a standard normal distribution. A negative skewness will lead to a distorted bell curve on the left, while positive skewness will cause a distorted bell curve on the right side.  What is Reinforcement Learning? It is an area of machine learning that takes the appropriate action to maximize returns for a given situation. Many software applications and machines use reinforcement learning (RL) to identify the best behavior/ path to arrive at the desired result for a specific situation.  Reinforcement learning is different from supervised learning. In the latter, the training data is the answer key to training the model with the correct answer. However, in RL, the reinforcement agent decides which task to perform, as there is no specific answer used for training. It allows machine learning developers to train the algorithm without using a dataset. The algorithm will learn from experience and improve itself over time.  What are the Different Datasets Available for Stock Market Predictions? Fortunately, there are a few datasets available to train the algorithms. Developers can access the datasets from the following:  NIFTY-50 Stock Market Data  The data is available from 1st January 2000 to 31st April 2021. It provides

Read More

Ankush Sharma on How AI is Revolutionizing Hiring – CEO Blindspots Podcast

🎧 Episode Overview In the latest episode of CEO Blindspots with Birgit Kamps, Ankush Sharma, the CEO of DataToBiz, shared his experience of developing and using HireLakeAI to successfully recruit and hire a talented team of professionals who contributed to the company’s recognition as one of the “Top 1000 fastest-growing firms”. Ankush explained how AI can streamline the recruitment process and help in selecting the most suitable candidates for each job role. Join Ankush Sharma as he shares the origin story of HireLakeAI and the efforts he and his co-founder, Parindsheel S Dhillon, put in during its initial phase. Discover how the COVID-19 pandemic impacted recruiting and HireLakeAI emerged as an innovative solution.  Tune in to the podcast! 💎 Key Takeaways 💬 Key Quotes “The DNA of a company is the reflection of the DNA of its founders.” “We do not believe in the culture of, hiring and firing.” 💡 To Sum Up! Join us in this episode as we explore how solving an internal challenge led to the creation of a game-changing solution for revolutionizing the hiring process. Discover the story behind HireLakeAI and the innovative ways it’s transforming the hiring culture.

Read More

How Connected Cars and Quantum Neural Network Can Help Drivers in Emergencies

Quantum neural networks are built based on quantum computing and classical physics to deliver accurate and reliable predictions. Our proposed model will make cars self-capable to handle emergencies. We’ll discuss the internal working and modules of our quantum neural network model for connected cars.  Quantum neural networks are created using the principles of quantum mechanics. These are typically feed-forward networks where the information collected in the previous layers is analyzed and forwarded to the next layer. Deep neural networks (DNNs) are already used in developing autonomous vehicles to define the right driving behavior for a vehicle. A Quantum neural network aims to assist drivers in effectively handling emergency situations.  Providing emergency support to car drivers using connected cars and quantum neural networks can reduce the risk of accidents and help drivers reach their destinations faster. This can potentially save lives, especially when driving to hospitals or emergency units. Quantum neural networks are more reliable and accurate than conventional neural networks.  Introduction to Connected Cars and Quantum Neural Network Model  A new system/ device will be embedded in the vehicle’s dashboard to provide support to drivers in emergency situations. The system offers second-to-second continuous support and is specifically designed to handle emergency or complex situations. The system collects data from the vehicle’s sensors and sends it to the connected cloud drive. This data will be processed using quantum neural networks built on the principles of quantum computing and classical physics. The concept of using quantum neural networks and connecting cars (through shared cloud data) is different from the existing approaches used in the industry. This model will be faster, reliable, accurate, and efficient enough to handle the worst-case scenarios people might experience when driving.  Resources Required for the Model  Data is the primary resource for this model. The quantum neural network model requires data from three sources to understand the situations, driving behavior, and the vehicle’s overall performance.  Descriptive Data  This data is about the car and its performance. The data is collected from sensors embedded in the engine, suspension, brake, tires (air pressure), etc. This data will be used to identify car’s health and quality. It also provides information about what’s happening in the car every second. The quantum neural network model will be able to provide a suitable solution when it knows the car’s strengths and limitations.  Navigational Data  This data is related to the routes, navigations, and trips you take in the car. The model collects data from maps to determine the current location, destination, route map, etc. It also gathers data from side impact detection sensors, blind spot detection sensors, cyclist and pedestrian detection sensors, etc., to pinpoint your exact current location.  Behavioral Data  Behavioral data deals with drivers’ performance and abilities. The data is extracted from the sensors embedded in the dashboard. Different sensors are used to collect data necessary for the quantum neural network model to understand the driver’s health and current condition. The sensors help determine who the driver is and suggest a solution according to their driving history (collected and stored in the connected cloud).  Heartbeat sensors, eye-tracking sensors, and fingerprint sensors on the steering wheel are used for data collection. Sensors that track the driving patterns are also used to determine the abilities of the driver. Workflow of the Proposed Model Working Process of the Quantum Neural Network Model  The entire proposed concept will have four steps or modules:   Each module has a definite purpose and streamlines the data flow within the model to arrive at the desired outcome. The second module is where the majority of the work happens. It is divided into three sub-modules. Let’s explore each module in detail.  1. Data Extraction As the name suggests, the data collected from multiple sensors in the car and stored in the cloud are extracted into the APIs. The process of collecting data from the car’s sensors and sending them to the connected cloud drive is continuous. The vast amounts of data are then directly sent to the APIs, where preprocessing occurs.  2. Data Preprocessing  The APIs transfer the data to preprocessing module, which has three sub-modules to prepare the data for analysis.  Data Cleaning  The first sub-module cleans the data extracted from the connected drive APIs. This is a necessary step to improve data quality and increase the accuracy of the quantum neural network model.  Naturally, data collected from multiple sensors will have issues such as wrong image frames, incompatible data formats, corrupt data values, incomplete/ missing data values, etc. This will affect the quality of the final outcome. This sub-module uses different techniques and tools to clean data and repair the wrong image frames. It tries to resolve the missing/ incomplete data or remove it totally. Statistical techniques are used to identify the issues with data and clean it accordingly.  Data Preprocessing  Preprocessing is similar to structuring and formatting data in large datasets. This sub-module prepares the cleaned data to make it ready for transformation, training, and predictions. The data is categorized based on its source.  For example, data from the cameras are sent to the video processing module. Data from heartbeat sensors go to the numerical processing module, and so on. New data categories will be created to sort the cleaned input data into neat segments/ types, making it easy for the quantum neural network to process.  Data Transformation  The last sub-module of the preprocessing stage is data transformation. Here, the preprocessed and sorted data is transformed to create a summary of what it contains. This helps understand the actual meaning of the data before it is fed into the quantum neural network for predictions. The transformed data is analyzed to arrive at the summary and is fed into the learning phase of the system.  3. Training and Predicting Outcomes using Quantum Neural Network Model  This module deals with training the quantum neural network to become capable of working with large datasets and delivering accurate predictions in less time. The data transformed in the previous module is fed

Read More

Dimensionality Reduction Techniques in Data Science

Dimensionality reduction techniques are basically a part of the data pre-processing step, performed before training the model. Analyzing data with a list of variables in machine learning requires a lot of resources and computations, not to mention the manual labor that goes with it. This is precisely where the dimensionality reduction techniques come into the picture. The dimensionality reduction technique is a process that transforms a high-dimensional dataset into a lower-dimensional dataset without losing the valuable properties of the original data. These dimensionality reduction techniques are basically a part of the data pre-processing step, performed before training the model. What is Dimensionality Reduction in Data Science? Imagine you are training a model that could predict the next day’s weather based on the various climatic conditions of the present day. The present-day conditions could be based on sunlight, humidity, cold, temperature, and many millions of such environmental features, which are too complex to analyze. Hence, we can lessen the number of features by observing which of them are strongly correlated with each other and clubbing them into one.  Here, we can club humidity and rainfall into a single dependent feature since we know they are strongly correlated. That’s it! This is how the dimensionality reduction technique is used to compress complex data into a simpler form without losing the essence of the data. Moreover, data science and AI experts are now also using data science solutions to leverage business ROI. Data visualization, data mining, predictive analytics, and other data analytics services by DataToBiz are changing the business game. Why is Dimensionality Reduction Necessary? machine learning and Deep Learning techniques are performed by inputting a vast amount of data to learn about fluctuations, trends, and patterns. Unfortunately, such huge data consists of many features, which often leads to a curse of dimensionality.  Moreover, sparsity is a common occurrence in large datasets. Sparsity refers to having negligible or no value features, and if it is inputted in a training model, it performs poorly on testing. In addition, such redundant features cause problems in clustering similar features of the data. Hence, to counter the curse of dimensionality, dimensionality reduction techniques come to the rescue. The answers to the question of why dimensionality reduction is useful are: Now let us understand which algorithms are used for dimensionality reduction of data with examples. What are the Dimensionality Reduction Techniques The dimensionality reduction techniques are broadly divided into two categories, namely, 1. Linear Methods PCA Principal Component Analysis (PCA) is one of the used DR techniques in data science. Consider a set of ‘p‘ variables that are correlated with each other. This technique reduces this set of ‘p‘ variables into a smaller number of uncorrelated variables, usually denoted by ‘k‘, where (k<p). These ‘k‘ variables are called principal components, and their variation is similar to the original dataset. PCA is used to figure out the correlation among features, which it combines together. As a result, the resultant dataset has lesser features that are linearly correlated with each other. This way, the model performs the reduction of correlated features while simultaneously calculating maximum variance in the original dataset. After finding the directions of this variance, it directs them into a smaller dimensional space which gives rise to new components called principal components. These components are pretty sufficient in representing the original features. Therefore, it reduces the reconstruction error while finding out the optimum components. This way, data is reduced, making the machine learning algorithms perform better and faster.  PrepAI is one of the perfect examples of AI that has made use of the PCA technique in the backend to generate questions from a given raw text intelligently. Factor Analysis This technique is an extension of Principal Component Analysis (PCA). The main focus of this technique is not just to reduce the dataset. It focuses more on finding out latent variables, which are results of other variables from the dataset. They are not measured directly in a single variable.  Latent variables are also called factors. Hence, the process of building a model which measures these latent variables is known as factor analysis. It not only helps in reducing the variables but also helps in distinguishing response clusters. For example, you have to build a model which will predict customer satisfaction. You will prepare a questionnaire that has questions like, “Are you happy with our product?” “Would you share your experience with your acquaintances?” If you want to create a variable to rate customer satisfaction, you will either average the responses or create a factor-dependent variable. This can be performed using PCA and keeping the first factor as a principal component. Linear Discriminant Analysis It is a dimensionality reduction technique that is used mainly for supervised classification problems. Logistic Regression fails in multi-classification. Hence, LDA comes into the picture to counter that shortcoming. It efficiently discriminates between training variables in their respective classes. Moreover, it is different from PCA as it calculates a linear combination between the input features to optimize the process of distinguishing different classes.  Here is an example to help you understand LDA: Consider a set of balls belonging to two classes: Red Balls and Blue Balls. Imagine they are plotted on a 2D plane randomly, such that they cannot be separated into two distinct classes using a straight line. In such cases, LDA is used, which can convert a 2D graph into a 1D graph, thereby maximizing the distinction between the classes of balls. The balls are projected to a new axis which separates them into their classes in the best possible way. The new axis is formed using two steps: SVD Consider data with ‘m‘ columns. Truncated Singular Value Decomposition method (TSVD) is a projection method where these ‘m‘ columns (features) are projected into a subspace with ‘m‘ or lesser columns without losing the characteristics of the data. An example where TSVD can be used is a dataset containing reviews about e-commerce products. The review column is mostly left blank, which gives rise to null values in the data, and TSVD tackles it efficiently.

Read More

Good life, Great life Podcast: Ankush Sharma on DataToBiz and Entrepreneurship

🎧 Episode Overview Ankush Sharma joined Brian Highfield in the 24th episode of Good Life, Great Life. Ankush shared his experiences starting his first entrepreneurship venture, DataToBiz, and how it changed him. A data evangelist by profession, Ankush, believes that data science is the answer to most complex business problems. Self-motivation defines Ankush’s career and is the foundation for DataToBiz.  Listen to Ankush Sharma talk about how DataToBiz came into existence and the efforts he and his co-founder Parindsheel S Dhillon put in during the initial phase. Find out what the COVID-19 pandemic brought to the table and where DataToBiz stands today.  💎 Key Takeaways:  💬 Key Quotes: “Hire people who are smarter than you so that you can delegate work and be confident that they’ll take care of it.”  “The work culture in a startup should be a reflection of the founder, and for me, people come first.” To Sum Up… This episode is about an IT employee-turned-entrepreneur’s journey in the competitive world. It’s not just about the first step, but the subsequent steps Ankush has to take to establish and successfully run DataToBiz, an AI & BI consulting company.  Attitude matters and Ankush Sharma shows why. 

Read More

Optimizing the T5 Model for Fast Inference

Model deployment is the most important aspect of developing a machine learning product. The development of every product starts with an idea and then we check if it’s feasible or not. Once we are sure that it is feasible we gather the data and start developing a model for it. During the model development, we also keep in mind the accuracy of the model, as well as its size and inference time as all of these factors, are very important for deploying the model but despite doing all these things sometimes we are settled with a model that has good accuracy but its inference time is not so good which might affect both the cost and user experience of the product hence such models needs to be optimized for inference as much as possible so that we can minimize the inference time to as low as possible. In this blog, we will explore the way by which we can optimize a T5 model for fast inference on a GPU.  Motivation Behind T5 Optimization We were using the T5 model in our product PrepAI which is a question generation platform where users can upload different kinds of documents, videos, or copy and paste texts and the platform automatically generates different kinds of questions with the provided content. Hence we decided to work on optimizing the T5 model. In this tutorial, we will consider the T5 model is trained for translation tasks and will do the comparison on translation tasks only. Optimization Using TensorRT As a developer whenever we think of optimizing a model for the Nvidia GPU, TensorRT is the obvious choice that comes to our mind because it transforms the model’s graph into a form that can benefit from the architectural structure of the GPU. It achieves this by replacing certain operations in a graph with other operations which are equivalent to this operation but have less computational overhead. It also fuses certain operations together into one and reduces the computational overhead this process is known as graph fusion. We thought of trying the TensorRT for optimizing our T5 model. For this, we used the same codes in the official Nvidia repo to implement the optimization as we just wanted to see how much performance can be achieved by using the implementation. We found out that the model was around 3-4x faster for smaller sequences like <100 tokens but after that, the model starts to slow down, and at 512 tokens long sequences it becomes slower than the original torch model. At that time according to certain developers, there was some bug with TensorRT which later got fixed later on when we switched to the next phase in optimization involving ONNX. Later on, we also found that the implementation also excludes the past key values caching which is essential for speedup on longer sequences. However, this can change in the future and TensorRT might beat other implementations but at the time of writing this blog, it hasn’t been implemented yet.  Optimization using ONNX runtime After trying out the TensorRT we decided to optimize the model with ONNX runtime. Converting any model to ONNX and applying little optimization automatically speed up the model by a small bit. During optimizing the model ONNX does basic operations like removing unused nodes, conversion of variables to constant, etc. For models like BERT, BART, GPT-2, Roberta, etc the ONNX also implements graph fusion which fuses the graphs of these models similar to the way TensorRT does. But unfortunately, the T5 model is not available yet. Since graph optimization was out of the plate the obvious method seemed to us was the conversion of the model into float16. Converting Encoder Into float16 The T5 model is an encoder-decoder model hence we tried to optimize the encoder first and then the decoder next. For doing this we utilized the ONNX runtime transformer optimization package. We first all the nodes of the ONNX encoder graph to float 16 and tried to evaluate the speed and accuracy of the model. We observed that converting all the nodes in the encoder destabilizes the encoder and hence the encoder only produces NAN values. The reason for this is that the encoder has a lot of operations which doesn’t work well in float16. Also, the T5 was never designed to be fully compatible with float16 but with bfloat16 and float32 data types. Hence we decided to identify those unstable nodes and keep them in float32 only doing that kept our model stable as well as improved the speed of the model. Conversion of Decoder Into float16 The next part is to optimize the decoder part of the model. Optimizing the decoder is more important than optimizing the encoder as in each generation the number of times an encoder run is for 1 time while the decoder runs for n times where n is the length of the target sentence. Similar to the encoder we started by converting all the nodes in the decoder part of the model into float16 and then evaluating to see how this has affected the accuracy and speed of the model we noticed that the model was not so fast for smaller sequences and it became slower for longer sequences. Digging deep into the ONNX runtime and some open source libraries we came to know about the cause behind this slowing. It was related to the data movement between the GPU memory and the RAM. We know that the decoder accepts the decoder_input_ids and the encoder_ouputs for generating the next token. Each time a prediction is made the input is transferred from RAM to GPU memory and after the calculation when the output which is in ORT format needs to be converted into NumPy which can only be done after moving the data back to CPU. The sizes of these tensors would be large. Moving these tensors consumes a lot of time and hence our model becomes slow as the size of the target output increases as

Read More

Analyzing the Brain Waves Data Using Python

The brain waves play a crucial role in sending signals to different parts of the body. Analyzing this data helps scientists uncover the intricacies and complexities of the human brain and provide solutions to help people with brain-related disorders. We’ll discuss brain waves and ways to use Python to analyze this data.  The human brain is a key organ to keep us functioning and active throughout our lives. We know that the concepts of artificial intelligence, deep learning, and artificial neural networks are derived from the working of the human brain. ANNs replicate the patterns and designs of the neural networks in the brain to allow the machines to perceive this and analyze data as the human brain does. This, in turn, helps doctors and scientists use advanced technology to understand the complexities of the human brain and identify activity related to brain diseases. We’ll deal with one such method to study the functioning of the brain and analyze the signals it sends to other parts of the body.  The brain constantly generates waves of electrical activity. The pattern of the waves changes based on the emotion a person experiences at that point. Devices like EEG kits are used to detect and record wave patterns. Computer programming languages like Python can be used to analyze this data. It helps determine how alert or focused a person is. By processing large datasets with such information, scientists can identify the causes/ reasons for brain diseases and find ways to cure them.  Let’s start by reading more about brain waves and the types before learning how Python helps analyze the brain wave data.  What are Brain Waves?  Brain waves are the electrical pulses used by neurons to communicate with each other. The neurons use electrical impulses to send signals about different human emotions and behaviors.  The frequency of each brain wave is different, depending on the emotion felt and displayed by the person. Measured in hertz (Hz) or cycles per second, we have slow and fast brain waves released by the neurons. The brain waves are given individual names to differentiate one from another based on frequency.  Types of Brain Waves There are five types of brain waves, with delta being the slowest and gamma being the fastest. The level of human awareness is determined by the frequency/ speed of the brain waves.  Delta Brain Waves  As the slowest of all, these high-amplitude brain waves have a frequency of 1 to 3 Hz and are experienced by humans when they are asleep.  Theta Brain Waves The Theta waves have a frequency range of 4 to 7 Hz and are found when a person is in a dreamy state. When the waves are close to the lower end, they represent the state when a person hovers between sleep and consciousness. It’s also known as the twilight state. Theta waves, in general, signify that mental inefficiency or that the person is either too relaxed or blanked out (zoned out) at that moment.  Alpha Brain Waves The alpha brain waves have a frequency range of 8 to 12 Hz. These are larger and slower, representing a relaxed or calm state of mind for a person ready to get into action if the need arises. The alpha brain waves are generated when someone feels peaceful after closing their eyes and picturing something they like.  Beta Brain Waves Beta brain waves are faster and smaller, with a frequency range of 13 to 38 Hz. These waves imply that the person is focused on something. They signify alertness, where the person is in their senses and displays all signs of concentration and mental activity.  Gamma Brain Waves Gamma brain waves are the fastest ones, with a frequency range of 38 to 42 Hz. These are subtle compared to the other brain waves and work on the consciousness and perception of the person. The waves occur when a person is highly alert and can feel every minute change in their surroundings.  Waveforms of Different Brain Waves: Capturing Brain Waves EEG (Electroencephalography) is a popular and most used method to capture brain waves and record the electrogram of the electrical activity on the scalp. It represents the macroscopic activity of the brain waves inside the brain. The electrodes are placed on the scalp to record the activity. EEG is typically a non-invasive process. However, Electrocorticography (Intracranial EEG) is an invasive process.  The method measures the fluctuations in the voltage of the ionic current released within the neurons. In clinical terms, EEG is the recording of spontaneous electrical activity in the brain over a period. This data is collected through the numerous electrodes placed on the scalp of a person.  The focus of the diagnosis is either on spectral content or the event-related potentials, during a particular duration or for a particular event. Spectral content, on the other hand, analyzes the type of neural oscillations or the brain waves.  Applications Used to Analyze Brain Waves  Two major applications analyze the brain waves, where each application focuses on a different aspect of analysis.  Emotion Analysis  Human emotions are determined by the brain. The brain waves carry messages with emotions that make the person feel something. This ‘emotion’ can be understood by analyzing brain waves. However, the concept of emotion and what it represents varies from one person to another based on cultural and environmental backgrounds. A classification algorithm is vital to accurately analyze the emotional aspect of brain waves.  Brain-Computer Interface (BCI)  The function of BCIs is to collect the brain waves, analyze the messages, translate the messages to commands and relay them to the output devices. BCIs don’t use neuromuscular pathways for this process. That’s because the purpose of using BCIs is to restore the functioning of the neuromuscular pathways in people suffering from brain diseases.  For example, cerebral palsy, stroke, amyotrophic lateral sclerosis, or spinal cord injury can damage the pathways and affect the transmission of brain waves. BCIs aim to restore the damage done so that the person can

Read More
Services
Data Engineering
Azure
AWS
GCP
Big Data
Data Warehousing
AI & Machine Learning
Artificial Intelligence
Natural Language Processing
Vision Analytics
Large Language Model (LLM)
Business Intelligence
PowerBI
Tableau

DataToBiz is a Data Science, AI, and BI Consulting Firm that helps Startups, SMBs and Enterprises achieve their future vision of sustainable growth.

Hire Developers
Get In Touch
Use Cases
Digital Transformation
AI Insurance Automation
Boutique Analytics
Ecommerce Analytics
Manufacturing Analytics
Digital Lending Automation
Sales Analytics
OEE Analytics
HR Analytics
Procurement Analytics
IT JIRA tracking
Products
PrepAI
Tally BI
Marketing Cockpit
Virtual Try On Platform
HirelakeAI
SensiblyAI
DataToBiz CV Platform
Quick Links
Careers
About
Partner Program
Knowledge Base
Case Studies
Industries

DataToBiz is a Data Science, AI, and BI Consulting Firm that helps Startups, SMBs and Enterprises achieve their future vision of sustainable growth.

Hire Developers
Get In Touch
DMCA.com Protection Status
Hire Developers
Request a Quote

Get a Free Data Analysis Done!

Need experts help with your data? Drop Your Query And Get a 30 Minutes Consultation at $0.

They have the experience and agility to understand what’s possible and deliver to our expectations.

GetMee

Founder & CEO

Drop Your Concern!