logo
episode-header-image
Dec 2017
20m 37s

[MINI] Parallel Algorithms

Kyle Polich
About this episode

When computers became commodity hardware and storage became incredibly cheap, we entered the era of so-call "big" data. Most definitions of big data will include something about not being able to process all the data on a single machine. Distributed computing is required for such large datasets.

Getting an algorithm to run on data spread out over a variety of different machines introduced new challenges for designing large-scale systems. First, there are concerns about the best strategy for spreading that data over many machines in an orderly fashion. Resolving ambiguity or disagreements across sources is sometimes required.

This episode discusses how such algorithms related to the complexity class NC.

Up next
Jul 27
Social Choice for Fair Recommendations
Recommender systems influence nearly every aspect of our digital lives—but what does it mean for those systems to be fair? Robin Burke joins Data Skeptic to discuss the history of recommender systems, the limitations of optimizing purely for accuracy, and how ideas from social ch ... Show More
42m 56s
Jul 2
News Recommendations
News recommendation algorithms influence far more than what stories we click—they can shape our understanding of the world. In this episode, Kyle Polich speaks with Andreea Iana about responsible AI, filter bubbles, multilingual news recommendation, and her open-source NewsRecLib ... Show More
46m 6s
Jun 23
Give Users the Wheel
What if you could simply tell a recommendation system what you want instead of relying on likes, dislikes, and watch history? Kyle Polich talks with Fuyuan Lyu about the DPR framework, which combines large language models and traditional recommender systems to give users direct c ... Show More
35m 28s
Recommended Episodes
Feb 2022
AI Today Podcast: Overview of Synthetic Data
Machine learning algorithms need examples of data from which they can learn, especially supervised machine learning algorithms. However, one big challenge for those looking to put machine learning into practice is the lack of a sufficient quantity of good quality data examples fr ... Show More
47m 14s
Feb 2024
A Small Episode About Big Data
What does Big Data actually mean? How has the science of Big Data changed recently? What are the potential benefits and pitfalls of Big Data? See omnystudio.com/listener for privacy information. 
39m 9s
Sep 2021
Massively Parallel Data Processing In Python Without The Effort Using Bodo
<div class="wp-block-jetpack-markdown"><h2>Summary</h2> <p>Python has beome the de facto language for working with data. That has brought with it a number of challenges having to do with the speed and scalability of working with large volumes of information.There have been many ... Show More
1h 4m
Mar 2024
The Y2K Bug, Part 1
In the 1950s and 60s - even leading into the 1990s - the cost of storage was so high, that using a 2-digit field for dates in a software instead of 4-digits could save an organization between $1.2-$2 Million dollars per GB of data. From this perspective, programming computers in ... Show More
27m 8s
Jul 2020
What data transformation library should I use? Pandas vs Dask vs Ray vs Modin vs Rapids (Ep. 112)
<p>In this episode I speak about data transformation frameworks available for the data scientist who writes Python code. The usual suspect is clearly Pandas, as the most widely used library and de-facto standard. However when data volumes increase and distributed algorithms are ... Show More
21m 10s
Nov 2022
Analyze Massive Data At Interactive Speeds With The Power Of Bitmaps Using FeatureBase
<div class="wp-block-jetpack-markdown"><h2>Summary</h2> <p>The most expensive part of working with massive data sets is the work of retrieving and processing the files that contain the raw information. FeatureBase (formerly Pilosa) avoids that overhead by converting the data int ... Show More
59m 25s
Jan 2021
Pyspark
Will clustered servers for big data drive machine learning pipelines 
15m 58s
Oct 2013
Bioinformatics: What the Clinical Laboratorian Needs to Know and Prepare For
New diagnostics technologies such as microarrays, next generation, or massively parallel sequencing, are generating an unprecedented amount of data. This requires a sophisticated knowledge of bioinformatics for proper storage, analysis, and mining of these very large data sets. 
13m 22s