Data Portability: NYPD vs. Palantir

The concept of data portability is all too often simplified to the question “Can we have our data in a machine readable format?”

At best, this ensures the data can be loaded to a new system and at worst the data can be added to a data warehouse.

Palantir, Peter Thiels company, provides a lens into the importance of “thinking from the end” with systems — i.e. what happens what the system potentially reaches obsolescence or you simply want to leave it for whatever reason.

Can you get the data out?

And, can you output the analysis in machine readable format to continue the work in another system?

I assume most companies will not hand over queries as they could claim they are proprietary.

However, this complexity, clear provides the importance of creating in-house capabilities for analytics, and building in the end of contract clauses and what they actually mean into contracts. 


The department has created a new system to replace Palantir, and it wants to transfer the analysis generated by Palantir’s software to the new system. But Palantir, the NYPD claims, has not produced the full analysis in a standardized format — one that would work with the new software — despite multiple requests from the police department in recent months.

Big data helped New York’s cops bust Bobby Shmurda. But as the NYPD’s contract with tech giant Palantir comes to an end, things could get messy.
— Read on www.buzzfeednews.com/article/williamalden/theres-a-fight-brewing-between-the-nypd-and-silicon-valley

Machine Learning Innovation Rules

It’s tempting to get caught in the trap of applying a promising new technology like Machine Learning to every problem, but try and resist this temptation. Instead, try to follow Joshua Porter’s advice as he states in his Principles of Product Design:

“…[P]roduct innovation isn’t about new products that solve new problems. Product innovation is about new products that solve existing problems better than they’re currently solved.”

https://spotify.design/articles/2019-12-10/three-principles-for-designing-ml-powered-products/

Deep Learning Basics

Deep Learning Basics

While reading Hung Lee’s Recruiting Brainfood, I stumbled upon this deep learning primer:

The Simple Guide to Deep Learning

The primer is great, and a quick read. Here is my quick summary below:

The basics of deep learning is to think about how the brain breaks up a specific task. For example, let’s say you are hiking the Appalachian Trail, and you see something in the distance running towards you. First, you might notice it is moving. Then, you might notice what shape it is. Then, you might notice how fast it is going. Then, you might notice a big snout. Then, your brain will determine that this is an animal.

The process would continue until your brain evaluated, classified and predicts what object it is seeing. The joy of the mental exercise (for me) is to understand how the human mind works to break down ideas.

Inputs > Algorithm > Prediction > Training: 

The following are the key concepts for thinking about deep learning concepts. Yes, this is overly simplified, but it is still a helpful start. 

  • Inputs: Labels/Images
  • Algorithm: 
    • Levels of Abstraction 1: Is this a shape?
    • Level of Abstraction 2: Is this shape an ear?
    • Level of Abstraction 3: Is this a cat?
  • Prediction = Yes or No. Is this prediction correct?

Current-State of Deep Learning:

  • Supervised Deep Learning: In effect, this is attempting to clone human behavior via labeled images, video, text or speech. 
  • Reinforcement Learning: This is where the model attempts to “learn” behaviors, codify those behaviors (i.e. what does that mean), and then implement strategies to optimize based on those strategies. As the article suggests, the following are some examples:
    • E-Commerce: model learns customer behaviors and tailors service to suit customer interests. 
    • Finance: model learns market behavior and generates trading strategies. 
    • Robots: model learns how physical world behaves (through video) and then navigates that world.

Network Architecture to Detect Objects in Images:

  • Input: Image
  • Extract Feature: Extract the specific features
  • Classification: Classify based on the probability of those features
  • Output: Image prediction

Enjoy your deep learning explorations!