ajet/df-research
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
88206d519cb4395953288ba474040a71e8f3ad29
df-research/tech.ml.dataset/docs/100-walkthrough.html
ajet bdf064f54d init research
2026-02-08 11:20:43 -10:00

1093 lines
62 KiB
HTML
Vendored
Raw Blame History

<!DOCTYPE html PUBLIC ""
"">
<html><head><meta charset="UTF-8" /><title>tech.ml.dataset Walkthrough</title><script async="true" src="https://www.googletagmanager.com/gtag/js?id=G-RGTB4J7LGP"></script><script>window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-95TVFC1FEB');</script><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="highlight/solarized-light.css" /><script type="text/javascript" src="highlight/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a> with <a href="https://github.com/xsc/codox-theme-rdash">RDash UI</a> theme</h2><h1><a href="index.html"><span class="project-title"><span class="project-name">TMD</span> <span class="project-version">8.003</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="000-getting-started.html"><div class="inner"><span>tech.ml.dataset Getting Started</span></div></a></li><li class="depth-1 current"><a href="100-walkthrough.html"><div class="inner"><span>tech.ml.dataset Walkthrough</span></div></a></li><li class="depth-1 "><a href="200-quick-reference.html"><div class="inner"><span>tech.ml.dataset Quick Reference</span></div></a></li><li class="depth-1 "><a href="columns-readers-and-datatypes.html"><div class="inner"><span>tech.ml.dataset Columns, Readers, and Datatypes</span></div></a></li><li class="depth-1 "><a href="nippy-serialization-rocks.html"><div class="inner"><span>tech.ml.dataset And nippy</span></div></a></li><li class="depth-1 "><a href="supported-datatypes.html"><div class="inner"><span>tech.ml.dataset Supported Datatypes</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tech</span></div></div></li><li class="depth-2"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>v3</span></div></div></li><li class="depth-3"><a href="tech.v3.dataset.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>dataset</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.categorical.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>categorical</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.clipboard.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>clipboard</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.column.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>column</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.column-filters.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>column-filters</span></div></a></li><li class="depth-4"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>io</span></div></div></li><li class="depth-5 branch"><a href="tech.v3.dataset.io.csv.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>csv</span></div></a></li><li class="depth-5 branch"><a href="tech.v3.dataset.io.datetime.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>datetime</span></div></a></li><li class="depth-5 branch"><a href="tech.v3.dataset.io.string-row-parser.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>string-row-parser</span></div></a></li><li class="depth-5"><a href="tech.v3.dataset.io.univocity.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>univocity</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.join.html"><div class="inner"><span class="tree" style="top: -145px;"><span class="top" style="height: 154px;"></span><span class="bottom"></span></span><span>join</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.math.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>math</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.metamorph.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>metamorph</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.modelling.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>modelling</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.print.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>print</span></div></a></li><li class="depth-4"><a href="tech.v3.dataset.reductions.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>reductions</span></div></a></li><li class="depth-5"><a href="tech.v3.dataset.reductions.apache-data-sketch.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>apache-data-sketch</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.rolling.html"><div class="inner"><span class="tree" style="top: -52px;"><span class="top" style="height: 61px;"></span><span class="bottom"></span></span><span>rolling</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.set.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>set</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.dataset.tensor.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tensor</span></div></a></li><li class="depth-4"><a href="tech.v3.dataset.zip.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>zip</span></div></a></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree" style="top: -641px;"><span class="top" style="height: 650px;"></span><span class="bottom"></span></span><span>libs</span></div></div></li><li class="depth-4 branch"><a href="tech.v3.libs.arrow.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>arrow</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.libs.clj-transit.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>clj-transit</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.libs.fastexcel.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>fastexcel</span></div></a></li><li class="depth-4"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>guava</span></div></div></li><li class="depth-5"><a href="tech.v3.libs.guava.cache.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>cache</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.libs.parquet.html"><div class="inner"><span class="tree" style="top: -52px;"><span class="top" style="height: 61px;"></span><span class="bottom"></span></span><span>parquet</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.libs.poi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>poi</span></div></a></li><li class="depth-4"><a href="tech.v3.libs.tribuo.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tribuo</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1>tech.ml.dataset Walkthrough</h1>
<p><code>tech.ml.dataset</code> (TMD) is a Clojure library designed to ease working with tabular data, similar to <code>data.table</code> in R or Python's Pandas. TMD takes inspiration from the design of those tools, but does not aim to copy their functionality. Instead, TMD is a building block that increases Clojure's already considerable data processing power.</p>
<h2>High Level Design</h2>
<p>In TMD, a dataset is logically a map of column name to column data. Column data is typed (e.g., a column of 16 bit integers, or a column of 64 bit floating point numbers), similar to a database. Column names may be any Java object - keywords and strings are typical - and column values may be any Java primitive type, or type supported by <code>tech.datatype</code>, datetimes, or arbitrary objects. Column data is stored contiguously in JVM arrays, and missing values are indicated with bitsets.</p>
<p>The outline of the walkthrough follows typical Clojure workflows, showing how those ideas are expanded by TMD:</p>
<ol>
<li><a href="#DatasetCreation">Dataset creation</a> from typical formats (csv, tsv, etc...), or sequences of maps, or maps of column name to column values</li>
<li><a href="#REPLPrinting">REPL-friendly printing</a> of datasets</li>
<li><a href="#AccessValues">Access to dataset (column) values</a>, and working with missing values</li>
<li><a href="#Selecting">Selecting</a> rows or columns of a dataset, or both at once</li>
<li><a href="#AddingRemovingUpdating">Adding, removing, and updating</a> columns</li>
<li><a href="#StatisticalAnalysis">Statistical analysis</a> of dataset data</li>
<li><a href="#SortFilterGroup">Functions familiar from <code>clojure.core</code></a> (e.g., <code>sort-by</code>, <code>filter</code>, <code>group-by</code>, etc...) that operate on datasets</li>
<li><a href="#ElementwiseOperations">Efficient elementwise operations</a> and their use in adding or updating columns derived from data in the dataset</li>
<li><a href="#GettingDataBackOut">Getting data back out</a> of datasets for downstream consumers</li>
</ol>
<hr />
<div id="DatasetCreation"></div>
<h2>Dataset Creation</h2>
<h3><code>-&gt;dataset</code>, <code>-&gt;&gt;dataset</code></h3>
<p>Creating a dataset from <code>.edn</code> (sequence of maps):</p>
<pre><code class="language-clojure">user&gt; (require '[tech.v3.dataset :as ds])
nil
user&gt; (ds/-&gt;dataset [{:a 1 :b 2} {:a 2 :c 3}])
_unnamed [2 3]:
| :a | :b | :c |
|----|----|----|
| 1 | 2 | |
| 2 | | 3 |
</code></pre>
<p>Passing options (as the first argument or the last, whichever is convenient):</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset [{:a 1 :b 2} {:a 2 :c 3}] {:dataset-name "simple"})
simple [2 3]:
| :a | :b | :c |
|---:|---:|---:|
| 1 | 2 | |
| 2 | | 3 |
user&gt; (ds/-&gt;&gt;dataset {:dataset-name "simple"} [{:a 1 :b 2} {:a 2 :c 3}])
simple [2 3]:
| :a | :b | :c |
|---:|---:|---:|
| 1 | 2 | |
| 2 | | 3 |
</code></pre>
<p>Creating a dataset from a file:</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset "test/data/stocks.csv")
test/data/stocks.csv [560 3]:
| symbol | date | price |
|--------|------------|-------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
| MSFT | 2000-06-01 | 32.54 |
| MSFT | 2000-07-01 | 28.40 |
| MSFT | 2000-08-01 | 28.40 |
| MSFT | 2000-09-01 | 24.53 |
| MSFT | 2000-10-01 | 28.02 |
| ... | ... | ... |
| AAPL | 2009-05-01 | 135.81 |
| AAPL | 2009-06-01 | 142.43 |
| AAPL | 2009-07-01 | 163.39 |
| AAPL | 2009-08-01 | 168.21 |
| AAPL | 2009-09-01 | 185.35 |
| AAPL | 2009-10-01 | 188.50 |
| AAPL | 2009-11-01 | 199.91 |
| AAPL | 2009-12-01 | 210.73 |
| AAPL | 2010-01-01 | 192.06 |
| AAPL | 2010-02-01 | 204.62 |
| AAPL | 2010-03-01 | 223.02 |
</code></pre>
<p><code>-&gt;dataset</code> and <code>-&gt;&gt;dataset</code> interpret a <code>String</code> arguments as a file path (URL), and the format is determined automatically on load. TMD has built-in broad support for typical formats.</p>
<p>For text-based formats such as csv and tsv, zipped (e.g., <code>*.csv.zip</code>) and gzipped (e.g., <code>*.tsv.gz</code>) inputs are uncompressed automatically on load.</p>
<p><code>InputStream</code>s also work instead of a <code>String</code>, this can be convenient when loading from private cloud storage.</p>
<p>While loading, TMD guesses at column datatypes, and generally does an ok job. If the guesses turn out to be sub-optimal, they can be overridden.</p>
<p>The same functions also load datasets from the internet:</p>
<pre><code class="language-clojure">user&gt; (def ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"))
#'user/ds
user&gt; (type ds)
tech.v3.dataset.impl.dataset.Dataset
user&gt; (ds/row-count ds)
1460
</code></pre>
<h4>Parsing Options</h4>
<p>The <code>-&gt;dataset</code> docstring documents the many options available for parsing:</p>
<ul>
<li><a href="https://techascent.github.io/tech.ml.dataset/tech.v3.dataset.html#var--.3Edataset">https://techascent.github.io/tech.ml.dataset/tech.v3.dataset.html#var--.3Edataset</a></li>
</ul>
<p>Limiting rows and columns at parse time:</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"
{:column-whitelist ["SalePrice" "1stFlrSF" "2ndFlrSF"]
:n-records 5})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 3]:
| SalePrice | 1stFlrSF | 2ndFlrSF |
|-----------|----------|----------|
| 208500 | 856 | 854 |
| 181500 | 1262 | 0 |
| 223500 | 920 | 866 |
| 140000 | 961 | 756 |
| 250000 | 1145 | 1053 |
</code></pre>
<p>Forcing a datatype for a column before parsing:</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"
{:column-whitelist ["SalePrice" "1stFlrSF" "2ndFlrSF"]
:n-records 5
:parser-fn {"SalePrice" :float32}})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [4 3]:
| SalePrice | 1stFlrSF | 2ndFlrSF |
|-----------|----------|----------|
| 208500.0 | 856 | 854 |
| 181500.0 | 1262 | 0 |
| 223500.0 | 920 | 866 |
| 140000.0 | 961 | 756 |
| 250000.0 | 1145 | 1053 |
</code></pre>
<p>Loading from <code>.xlsx</code>:</p>
<pre><code class="language-clojure">user&gt; (require '[tech.v3.libs.fastexcel]) ;; for xlsx support
nil
user&gt; (def ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx"))
#'user/ds
user&gt; ds
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx [1000 8]:
| column-0 | First Name | Last Name | Gender | Country | Age | Date | Id |
|---------:|------------|------------|--------|---------------|-----:|------------|-------:|
| 1.0 | Dulce | Abril | Female | United States | 32.0 | 15/10/2017 | 1562.0 |
| 2.0 | Mara | Hashimoto | Female | Great Britain | 25.0 | 16/08/2016 | 1582.0 |
| 3.0 | Philip | Gent | Male | France | 36.0 | 21/05/2015 | 2587.0 |
| 4.0 | Kathleen | Hanner | Female | United States | 25.0 | 15/10/2017 | 3549.0 |
| 5.0 | Nereida | Magwood | Female | United States | 58.0 | 16/08/2016 | 2468.0 |
| 6.0 | Gaston | Brumm | Male | United States | 24.0 | 21/05/2015 | 2554.0 |
| 7.0 | Etta | Hurn | Female | Great Britain | 56.0 | 15/10/2017 | 3598.0 |
| 8.0 | Earlean | Melgar | Female | United States | 27.0 | 16/08/2016 | 2456.0 |
| 9.0 | Vincenza | Weiland | Female | United States | 40.0 | 21/05/2015 | 6548.0 |
| 10.0 | Fallon | Winward | Female | Great Britain | 28.0 | 16/08/2016 | 5486.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 990.0 | Chase | Karner | Male | United States | 37.0 | 15/10/2017 | 2154.0 |
| 991.0 | Tommie | Underdahl | Male | United States | 26.0 | 16/08/2016 | 3265.0 |
| 992.0 | Dorcas | Darity | Female | United States | 37.0 | 21/05/2015 | 8765.0 |
| 993.0 | Angel | Sanor | Male | France | 24.0 | 15/10/2017 | 3259.0 |
| 994.0 | Willodean | Harn | Female | United States | 39.0 | 16/08/2016 | 3567.0 |
| 995.0 | Weston | Martina | Male | United States | 26.0 | 21/05/2015 | 6540.0 |
| 996.0 | Roma | Lafollette | Female | United States | 34.0 | 15/10/2017 | 2654.0 |
| 997.0 | Felisa | Cail | Female | United States | 28.0 | 16/08/2016 | 6525.0 |
| 998.0 | Demetria | Abbey | Female | United States | 32.0 | 21/05/2015 | 3265.0 |
| 999.0 | Jeromy | Danz | Male | United States | 39.0 | 15/10/2017 | 3265.0 |
| 1000.0 | Rasheeda | Alkire | Female | United States | 29.0 | 16/08/2016 | 6125.0 |
</code></pre>
<p>Notably, here, the "Date" column ended up as a <code>:string</code> due to the funny <code>dd/MM/yyyy</code> format:</p>
<pre><code class="language-clojure">user&gt; (meta (get ds "Date"))
{:categorical? true, :name "Date", :datatype :string, :n-elems 1000}
</code></pre>
<p>This can be overridden at parse time to get more robust types:</p>
<pre><code class="language-clojure">user&gt; (def ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx"
{:parser-fn {"Date" [:local-date "dd/MM/yyyy"]}}))
#'user/ds
user&gt; ds
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx [1000 8]:
| column-0 | First Name | Last Name | Gender | Country | Age | Date | Id |
|---------:|------------|------------|--------|---------------|-----:|------------|-------:|
| 1.0 | Dulce | Abril | Female | United States | 32.0 | 2017-10-15 | 1562.0 |
| 2.0 | Mara | Hashimoto | Female | Great Britain | 25.0 | 2016-08-16 | 1582.0 |
| 3.0 | Philip | Gent | Male | France | 36.0 | 2015-05-21 | 2587.0 |
| 4.0 | Kathleen | Hanner | Female | United States | 25.0 | 2017-10-15 | 3549.0 |
| 5.0 | Nereida | Magwood | Female | United States | 58.0 | 2016-08-16 | 2468.0 |
| 6.0 | Gaston | Brumm | Male | United States | 24.0 | 2015-05-21 | 2554.0 |
| 7.0 | Etta | Hurn | Female | Great Britain | 56.0 | 2017-10-15 | 3598.0 |
| 8.0 | Earlean | Melgar | Female | United States | 27.0 | 2016-08-16 | 2456.0 |
| 9.0 | Vincenza | Weiland | Female | United States | 40.0 | 2015-05-21 | 6548.0 |
| 10.0 | Fallon | Winward | Female | Great Britain | 28.0 | 2016-08-16 | 5486.0 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 990.0 | Chase | Karner | Male | United States | 37.0 | 2017-10-15 | 2154.0 |
| 991.0 | Tommie | Underdahl | Male | United States | 26.0 | 2016-08-16 | 3265.0 |
| 992.0 | Dorcas | Darity | Female | United States | 37.0 | 2015-05-21 | 8765.0 |
| 993.0 | Angel | Sanor | Male | France | 24.0 | 2017-10-15 | 3259.0 |
| 994.0 | Willodean | Harn | Female | United States | 39.0 | 2016-08-16 | 3567.0 |
| 995.0 | Weston | Martina | Male | United States | 26.0 | 2015-05-21 | 6540.0 |
| 996.0 | Roma | Lafollette | Female | United States | 34.0 | 2017-10-15 | 2654.0 |
| 997.0 | Felisa | Cail | Female | United States | 28.0 | 2016-08-16 | 6525.0 |
| 998.0 | Demetria | Abbey | Female | United States | 32.0 | 2015-05-21 | 3265.0 |
| 999.0 | Jeromy | Danz | Male | United States | 39.0 | 2017-10-15 | 3265.0 |
| 1000.0 | Rasheeda | Alkire | Female | United States | 29.0 | 2016-08-16 | 6125.0 |
user&gt; (meta (get ds "Date"))
{:name "Date", :datatype :local-date, :n-elems 1000}
user&gt; (first (get ds "Date"))
#object[java.time.LocalDate 0x36b45eff "2017-10-15"]
</code></pre>
<p>In general, supplying tuples of <code>[datatype parse-fn]</code> allows overriding. As shown above, for datetime types, <code>parse-fn</code> can additionally be a DateTimeFormat format string or a DateTimeFormat object.</p>
<p>A more complete example (necessary, because excel is lax with serialized types):</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx"
{:parser-fn {"Date" [:local-date "dd/MM/yyyy"]
"Id" :int32
"column-0" :int32
"Age" :int16}})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/file_example_XLSX_1000.xlsx [1000 8]:
| column-0 | First Name | Last Name | Gender | Country | Age | Date | Id |
|---------:|------------|------------|--------|---------------|----:|------------|-----:|
| 1 | Dulce | Abril | Female | United States | 32 | 2017-10-15 | 1562 |
| 2 | Mara | Hashimoto | Female | Great Britain | 25 | 2016-08-16 | 1582 |
| 3 | Philip | Gent | Male | France | 36 | 2015-05-21 | 2587 |
| 4 | Kathleen | Hanner | Female | United States | 25 | 2017-10-15 | 3549 |
| 5 | Nereida | Magwood | Female | United States | 58 | 2016-08-16 | 2468 |
| 6 | Gaston | Brumm | Male | United States | 24 | 2015-05-21 | 2554 |
| 7 | Etta | Hurn | Female | Great Britain | 56 | 2017-10-15 | 3598 |
| 8 | Earlean | Melgar | Female | United States | 27 | 2016-08-16 | 2456 |
| 9 | Vincenza | Weiland | Female | United States | 40 | 2015-05-21 | 6548 |
| 10 | Fallon | Winward | Female | Great Britain | 28 | 2016-08-16 | 5486 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 990 | Chase | Karner | Male | United States | 37 | 2017-10-15 | 2154 |
| 991 | Tommie | Underdahl | Male | United States | 26 | 2016-08-16 | 3265 |
| 992 | Dorcas | Darity | Female | United States | 37 | 2015-05-21 | 8765 |
| 993 | Angel | Sanor | Male | France | 24 | 2017-10-15 | 3259 |
| 994 | Willodean | Harn | Female | United States | 39 | 2016-08-16 | 3567 |
| 995 | Weston | Martina | Male | United States | 26 | 2015-05-21 | 6540 |
| 996 | Roma | Lafollette | Female | United States | 34 | 2017-10-15 | 2654 |
| 997 | Felisa | Cail | Female | United States | 28 | 2016-08-16 | 6525 |
| 998 | Demetria | Abbey | Female | United States | 32 | 2015-05-21 | 3265 |
| 999 | Jeromy | Danz | Male | United States | 39 | 2017-10-15 | 3265 |
| 1000 | Rasheeda | Alkire | Female | United States | 29 | 2016-08-16 | 6125 |
</code></pre>
<h4>Map of Columns Format</h4>
<p>Creating a dataset from a map of column name to column data:</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset {:name ["fred" "ethel" "sally"]
:age [42 42 21]
:likes ["pizza" "sushi" "opera"]})
_unnamed [3 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
</code></pre>
<p>TMD interprets untyped column data (like a persistent vector of objects for example) as either <code>string</code> or <code>double</code> depending on the first entry of the column data sequence. In contrast, typed column data (e.g., a Java array of some primitive type, or a typed <code>tech.v3.datatype</code> container) undergoes no interpretation.</p>
<hr />
<div id="REPLPrinting"></div>
<h2>REPL Friendly Printing</h2>
<p>In general, it is safe to print out very large datasets or columns. By default, TMD will only print out around twenty values, eliding with ellipses as necessary. This turns out to be super-useful, for example when getting a feel for the data in a particular dataset or column.</p>
<p>By default, datasets with more than 20 rows print abbreviated:</p>
<pre><code class="language-clojure">user&gt; (ds/-&gt;dataset (for [i (range 21)] {:a (rand)}))
_unnamed [21 1]:
| :a |
|-----------:|
| 0.61372362 |
| 0.17076240 |
| 0.99000549 |
| 0.75449856 |
| 0.95344183 |
| 0.06785647 |
| 0.85751960 |
| 0.29387977 |
| 0.12027818 |
| 0.31456890 |
| ... |
| 0.08843911 |
| 0.56787410 |
| 0.79682279 |
| 0.56327258 |
| 0.32338803 |
| 0.95067985 |
| 0.74832399 |
| 0.10802058 |
| 0.80990261 |
| 0.81393921 |
| 0.42608164 |
</code></pre>
<p>The helpful <code>print-all</code> function overrides this so every row is printed:</p>
<pre><code class="language-clojure">user&gt; (-&gt; (ds/-&gt;dataset (for [i (range 21)] {:a (rand)}))
(ds/print-all))
_unnamed [21 1]:
| :a |
|-----------:|
| 0.42679967 |
| 0.21603676 |
| 0.62454379 |
| 0.20501915 |
| 0.01175086 |
| 0.33983128 |
| 0.25639060 |
| 0.08791288 |
| 0.35557448 |
| 0.65832242 |
| 0.70054068 |
| 0.23177467 |
| 0.84374634 |
| 0.52497340 |
| 0.13117490 |
| 0.87235367 |
| 0.69944234 |
| 0.71521474 |
| 0.13842164 |
| 0.41571045 |
| 0.36734680 |
</code></pre>
<p>Note that <code>print-all</code> does not actually print anything, it merely changes the metadata on the dataset to override the default (abbreviated) printing behavior.</p>
<hr />
<p>Multiline printing:</p>
<pre><code class="language-clojure">user&gt; (require '[tech.v3.tensor :as dtt]) ;; Has some types that print nicely on multiple lines
nil
user&gt; (def t (dtt/-&gt;tensor (partition 3 (range 9))))
#'user/t
user&gt; (ds/-&gt;dataset [{:a 1 :b t}{:a 2 :b t}])
_unnamed [2 2]:
| :a | :b |
|---:|------------------------------|
| 1 | #tech.v3.tensor&lt;object&gt;[3 3] |
| | [[0 1 2] |
| | [3 4 5] |
| | [6 7 8]] |
| 2 | #tech.v3.tensor&lt;object&gt;[3 3] |
| | [[0 1 2] |
| | [3 4 5] |
| | [6 7 8]] |
</code></pre>
<p>Overriding this via metadata:</p>
<pre><code class="language-clojure">user&gt; (def ds *1)
#'user/ds
user&gt; (with-meta ds
(assoc (meta ds) :print-line-policy :single))
_unnamed [2 2]:
| :a | :b |
|---:|------------------------------|
| 1 | #tech.v3.tensor&lt;object&gt;[3 3] |
| 2 | #tech.v3.tensor&lt;object&gt;[3 3] |
</code></pre>
<p>Especially useful when dealing with datasets that may have large amounts of per-column data:</p>
<pre><code class="language-clojure">user&gt; (def events-ds (-&gt; (ds/-&gt;dataset "https://api.github.com/events"
{:key-fn keyword
:file-type :json})
(vary-meta assoc
:print-line-policy :single
:print-column-max-width 25)))
#'user/events-ds
user&gt; (ds/head events-ds)
https://api.github.com/events [5 8]:
| :id | :type | :actor | :repo | :payload | :public | :created_at | :org |
|-------------|------------------------|----------------|-----------------|-----------------------|---------|----------------------|----------------|
| 13911736787 | PushEvent | {:id 29139614, | {:id 253259114, | {:push_id 5888739305, | true | 2020-10-20T17:49:36Z | |
| 13911736794 | IssuesEvent | {:id 47793873, | {:id 240806054, | {:action "opened", | true | 2020-10-20T17:49:36Z | {:id 61098177, |
| 13911736759 | PushEvent | {:id 71535163, | {:id 304746399, | {:push_id 5888739282, | true | 2020-10-20T17:49:36Z | |
| 13911736795 | PullRequestReviewEvent | {:id 47063667, | {:id 305218173, | {:action "created", | true | 2020-10-20T17:49:36Z | |
| 13911736760 | PushEvent | {:id 22623307, | {:id 287289752, | {:push_id 5888739280, | true | 2020-10-20T17:49:36Z | |
</code></pre>
<h3>Markdown</h3>
<p>Output is printed as markdown tables:</p>
<p>So, this:</p>
<pre><code>| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
</code></pre>
<p>Renders nicely under markdown processing:</p>
<table>
<thead>
<tr><th>:name</th><th align="right">:age</th><th>:likes</th></tr>
</thead>
<tbody>
<tr><td>fred</td><td align="right">42</td><td>pizza</td></tr>
<tr><td>ethel</td><td align="right">42</td><td>sushi</td></tr>
<tr><td>sally</td><td align="right">21</td><td>opera</td></tr>
</tbody>
</table>
<p>The full list of possible printing options is provided in the documentation for <a href="https://techascent.github.io/tech.ml.dataset/tech.v3.dataset.print.html">dataset-data-&gt;str</a>.</p>
<hr />
<div id="AccessValues"></div>
<h2>Access To Dataset (Column) Values</h2>
<h3>Datasets are Maps</h3>
<p>Every Dataset implements <code>clojure.lang.IPersistentMap</code>. However, unlike Clojure's persistent maps, dataset columns are ordered.</p>
<p>Investigating a dataset as a map, and locating missing values:</p>
<pre><code class="language-clojure">user&gt; (def ds (ds/-&gt;dataset [{:a 1 :b 2} {:a 2 :c 3}]))
#'user/ds
user&gt; ds
_unnamed [2 3]:
| :a | :b | :c |
|---:|---:|---:|
| 1 | 2 | |
| 2 | | 3 |
user&gt; (first ds)
[:a #tech.v3.dataset.column&lt;int64&gt;[2]
:a
[1, 2]]
user&gt; (second ds)
[:b #tech.v3.dataset.column&lt;int64&gt;[2]
:b
[2, ]]
user&gt; (ds :b)
#tech.v3.dataset.column&lt;int64&gt;[2]
:b
[2, ]
user&gt; (ds/missing (ds :b))
{1}
user&gt; (ds/column-names ds)
(:a :b :c)
user&gt; (for [k (ds/column-names ds)]
[k (ds/missing (ds k))])
([:a {}] [:b {1}] [:c {0}])
</code></pre>
<h3>Columns are Indexed</h3>
<p>Columns implement <code>clojure.lang.Indexed</code> (and so provide nth) and also implement <code>clojure.lang.IFn</code> just like Clojure's persistent vectors.</p>
<p>Getting values is easy:</p>
<pre><code class="language-clojure">user&gt; (def lucy-ds (ds/-&gt;dataset {:name ["fred" "ethel" "sally"]
:age [42 42 21]
:likes ["pizza" "sushi" "opera"]}))
#'user/lucy-ds
user&gt; (lucy-ds :name)
#tech.v3.dataset.column&lt;string&gt;[3]
:name
[fred, ethel, sally]
user&gt; (nth (lucy-ds :name) 0)
"fred"
user&gt; ((lucy-ds :name) 0)
"fred"
user&gt; ((lucy-ds :name) 2)
"sally"
</code></pre>
<h3>As Clojure Data</h3>
<p>Accessing a dataset's rows as a sequence of maps (with one key per column):</p>
<pre><code class="language-clojure">user&gt; (ds/rows lucy-ds)
[{:name "fred", :age 42, :likes "pizza"}
{:name "ethel", :age 42, :likes "sushi"}
{:name "sally", :age 21, :likes "opera"}]
</code></pre>
<p>Accessing a dataset's rows as a sequence of vectors (with one entry per column):</p>
<pre><code class="language-clojure">user&gt; (ds/rowvecs lucy-ds)
[["fred" 42 "pizza"]
["ethel" 42 "sushi"]
["sally" 21 "opera"]]
</code></pre>
<p>These are just <em>views</em> on the dataset, and so accessing these does not double RAM usage, and a Clojureist's intuition about immutability, persistence, and structural sharing are respected (when possible).</p>
<hr />
<div id="Selecting"></div>
<h2>Selecting Dataset Rows or Columns - (Subrect Selection)</h2>
<p>A dataset with many columns:</p>
<pre><code class="language-clojure">user&gt; (def ames-ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz"))
#'user/ames-ds
user&gt; (ds/column-count ames-ds)
81
user&gt; (ds/column-names ames-ds)
("Id"
"MSSubClass"
"MSZoning"
...)
user&gt; (get ames-ds "OverallQual")
#tech.v3.dataset.column&lt;int16&gt;[1460]
OverallQual
[7, 6, 7, 7, 8, 5, 8, 7, 7, 5, 5, 9, 5, 7, 6, 7, 6, 4, 5, 5...]
</code></pre>
<p>Selecting - and implicitly reordering (!) - columns:</p>
<pre><code class="language-clojure">user&gt; (ds/select-columns ames-ds ["Id" "OverallQual" "SalePrice"])
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [1460 3]:
| Id | OverallQual | SalePrice |
|-----:|------------:|----------:|
| 1 | 7 | 208500 |
| 2 | 6 | 181500 |
| 3 | 7 | 223500 |
| 4 | 7 | 140000 |
| 5 | 8 | 250000 |
| 6 | 5 | 143000 |
| 7 | 8 | 307000 |
| 8 | 7 | 200000 |
| 9 | 7 | 129900 |
| 10 | 5 | 118000 |
| ... | ... | ... |
| 1450 | 5 | 92000 |
| 1451 | 5 | 136000 |
| 1452 | 8 | 287090 |
| 1453 | 5 | 145000 |
| 1454 | 5 | 84500 |
| 1455 | 7 | 185000 |
| 1456 | 6 | 175000 |
| 1457 | 6 | 210000 |
| 1458 | 7 | 266500 |
| 1459 | 5 | 142125 |
| 1460 | 5 | 147500 |
</code></pre>
<p>Interest in just a few specific rows:</p>
<pre><code class="language-clojure">user&gt; (-&gt; (ds/select-columns ames-ds ["Id" "OverallQual" "SalePrice"])
(ds/select-rows [0 1 3 5 17 42]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [6 3]:
| Id | OverallQual | SalePrice |
|---:|------------:|----------:|
| 1 | 7 | 208500 |
| 2 | 6 | 181500 |
| 4 | 7 | 140000 |
| 6 | 5 | 143000 |
| 18 | 4 | 90000 |
| 43 | 5 | 144000 |
</code></pre>
<p>Or the same as a one-liner:</p>
<pre><code class="language-clojure">user&gt; (ds/select ames-ds ["Id" "OverallQual" "SalePrice"] [0 1 3 5 17 42])
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [6 3]:
| Id | OverallQual | SalePrice |
|---:|------------:|----------:|
| 1 | 7 | 208500 |
| 2 | 6 | 181500 |
| 4 | 7 | 140000 |
| 6 | 5 | 143000 |
| 18 | 4 | 90000 |
| 43 | 5 | 144000 |
</code></pre>
<hr />
<div id="AddingRemovingUpdating"></div>
<h2>Adding, Removing, and Updating Columns</h2>
<p>Because datasets are maps (of column name to column data), adding a sequence as a column is trivial:</p>
<pre><code class="language-clojure">user&gt; (assoc lucy-ds :foo (repeatedly #(rand)))
_unnamed [3 4]:
| :name | :age | :likes | :foo |
|-------|-----:|--------|-----------:|
| fred | 42 | pizza | 0.21302376 |
| ethel | 42 | sushi | 0.40027647 |
| sally | 21 | opera | 0.04117065 |
</code></pre>
<p>Similarly, removing columns is just <code>dissoc</code>:</p>
<pre><code class="language-clojure">user&gt; (dissoc lucy-ds :age)
_unnamed [3 2]:
| :name | :likes |
|-------|--------|
| fred | pizza |
| ethel | sushi |
| sally | opera |
</code></pre>
<p>Look at these last two examples again, they subtly indicate that TMD datasets are <em>functional</em>... Neither operation changed <code>lucy-ds</code>.</p>
<p>🕉</p>
<hr />
<p>The powerful <code>row-map</code> function is often the best way to add (or update) columns derived from data already in the dataset:</p>
<pre><code class="language-clojure">user&gt; (-&gt; (ds/select ames-ds ["Id" "OverallQual" "SalePrice"] [0 1 3 5 17 42])
(ds/row-map (fn [{:strs [OverallQual SalePrice]}]
{"PricePerQual" (quot SalePrice OverallQual)
"BetterPrice" (* 2 SalePrice)})))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [6 5]:
| Id | OverallQual | SalePrice | PricePerQual | BetterPrice |
|---:|------------:|----------:|-------------:|------------:|
| 1 | 7 | 208500 | 29785 | 417000 |
| 2 | 6 | 181500 | 30250 | 363000 |
| 4 | 7 | 140000 | 20000 | 280000 |
| 6 | 5 | 143000 | 28600 | 286000 |
| 18 | 4 | 90000 | 22500 | 180000 |
| 43 | 5 | 144000 | 28800 | 288000 |
</code></pre>
<p>Another example of adding a column:</p>
<pre><code class="language-clojure">user&gt; (require '[tech.v3.datatype.functional :as dfn]) ;; for functional column operation
nil
user&gt; lucy-ds
_unnamed [3 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
user&gt; (dfn/log (:age lucy-ds))
[3.7376696182833684 3.7376696182833684 3.044522437723423]
user&gt; (assoc lucy-ds :log-age (dfn/log (:age lucy-ds)))
_unnamed [3 4]:
| :name | :age | :likes | :log-age |
|-------|-----:|--------|-----------:|
| fred | 42 | pizza | 3.73766962 |
| ethel | 42 | sushi | 3.73766962 |
| sally | 21 | opera | 3.04452244 |
</code></pre>
<p>Constants are broadcast to each row:</p>
<pre><code class="language-clojure">user&gt; (assoc lucy-ds :spring-chicken false)
_unnamed [3 4]:
| :name | :age | :likes | :spring-chicken |
|-------|-----:|--------|-----------------|
| fred | 42 | pizza | false |
| ethel | 42 | sushi | false |
| sally | 21 | opera | false |
</code></pre>
<hr />
<div id="StatisticalAnalysis"></div>
<h2>Statistical Analysis</h2>
<p>The <code>tech.v3.datatype.statistics</code> namespace from the underlying <code>dtype-next</code> system provides many useful statistical functions:</p>
<pre><code class="language-clojure">user&gt; (require '[tech.v3.datatype.statistics :as stats])
nil
user&gt; (def stock-ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"
{:key-fn keyword}))
#'user/stock-ds
user&gt; (ds/filter-column stock-ds :symbol #{"MSFT"})
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [123 3]:
| :symbol | :date | :price |
|---------|------------|-------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
| MSFT | 2000-06-01 | 32.54 |
| MSFT | 2000-07-01 | 28.40 |
| MSFT | 2000-08-01 | 28.40 |
| MSFT | 2000-09-01 | 24.53 |
| MSFT | 2000-10-01 | 28.02 |
| ... | ... | ... |
| MSFT | 2009-05-01 | 20.59 |
| MSFT | 2009-06-01 | 23.42 |
| MSFT | 2009-07-01 | 23.18 |
| MSFT | 2009-08-01 | 24.43 |
| MSFT | 2009-09-01 | 25.49 |
| MSFT | 2009-10-01 | 27.48 |
| MSFT | 2009-11-01 | 29.27 |
| MSFT | 2009-12-01 | 30.34 |
| MSFT | 2010-01-01 | 28.05 |
| MSFT | 2010-02-01 | 28.67 |
| MSFT | 2010-03-01 | 28.80 |
user&gt; (def prices (:price (ds/filter-column stock-ds :symbol #{"MSFT"})))
#'user/prices
user&gt; (stats/min prices)
15.81
user&gt; (stats/mean prices)
24.736747967479673
user&gt; (stats/max prices)
43.22
</code></pre>
<p>The <code>descriptive-stats</code> function calculates many statistics at once, returning the result as a dataset:</p>
<pre><code class="language-clojure">user&gt; (ds/descriptive-stats (ds/filter-column stock-ds :symbol #{"MSFT"}))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| :symbol | :string | 123 | 0 | | | MSFT | | | | MSFT | MSFT |
| :date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| :price | :float64 | 123 | 0 | 15.81 | 24.74 | | 43.22 | 4.30395786E+00 | 1.16559225 | 39.81 | 28.80 |
</code></pre>
<p>The same data is available as a sequence of maps (with one map per column) from <code>brief</code>:</p>
<pre><code class="language-clojure">user&gt; (ds/brief (ds/filter -column stock-ds :symbol #{"MSFT"}))
({:n-missing 0,
:col-name :symbol,
:histogram (["MSFT" 123]),
:datatype :string,
:mode "MSFT",
:n-valid 123,
:values ["MSFT"],
:first "MSFT",
:last "MSFT",
:n-values 1}
{:min #object[java.time.LocalDate 0x3059f98a "2000-01-01"],
:n-missing 0,
:col-name :date,
:mean #object[java.time.LocalDate 0x78193b42 "2005-01-30"],
:datatype :packed-local-date,
:skew 2.5335340939019964E-4,
:standard-deviation 9.375545376336331E10,
:quartile-3 #object[java.time.LocalDate 0x2a73389f "2007-09-01"],
:n-valid 123,
:quartile-1 #object[java.time.LocalDate 0x5818d8e "2002-07-01"],
:median #object[java.time.LocalDate 0x3eaaea8d "2005-02-01"],
:max #object[java.time.LocalDate 0x4aeeb094 "2010-03-01"],
:first #object[java.time.LocalDate 0x16291c62 "2000-01-01"],
:last #object[java.time.LocalDate 0x56533361 "2010-03-01"]}
{:min 15.81,
:n-missing 0,
:col-name :price,
:mean 24.736747967479673,
:datatype :float64,
:skew 1.165592245374292,
:standard-deviation 4.303957861320731,
:quartile-3 27.34,
:n-valid 123,
:quartile-1 21.75,
:median 24.11,
:max 43.22,
:first 39.81,
:last 28.8})
</code></pre>
<p>Supremely useful for developing generalized systems that work on arbitrary datasets.</p>
<hr />
<div id="SortFilterGroup"></div>
<h2>Sort-by, Filter, Group-by</h2>
<p>While conceptually similar to the ones in <code>clojure.core</code>, these functions are aware of the tabular/columnar dataset structure, and so can be much more efficient (especially when using the single-column versions).</p>
<p>When using the whole-dataset versions, TMD passes each row (as a map) into the argument function, this matches the expectation of Clojure programmers:</p>
<pre><code class="language-clojure">user&gt; (-&gt; ames-ds
(ds/filter #(&lt; 30000 (get % "SalePrice")))
(ds/select ["OverallQual" "SalePrice"] (range 5)))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 2]:
| OverallQual | SalePrice |
|------------:|----------:|
| 7 | 208500 |
| 6 | 181500 |
| 7 | 223500 |
| 7 | 140000 |
| 8 | 250000 |
</code></pre>
<p>Operating on a single column works the same, and is faster, so do that when possible:</p>
<pre><code class="language-clojure">user&gt; (-&gt; ames-ds
(ds/filter-column "SalePrice" #(&lt; 30000 %))
(ds/select ["OverallQual" "SalePrice"] (range 5)))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 2]:
| OverallQual | SalePrice |
|------------:|----------:|
| 7 | 208500 |
| 6 | 181500 |
| 7 | 223500 |
| 7 | 140000 |
| 8 | 250000 |
</code></pre>
<h3>Sorting</h3>
<p>Sorting (again, like in Clojure):</p>
<pre><code class="language-clojure">user&gt; (-&gt; (ds/sort-by-column ames-ds "SalePrice" &gt;)
(ds/select-columns ["Id" "OverallQual" "SalePrice"]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [1460 3]:
| Id | OverallQual | SalePrice |
|-----:|------------:|----------:|
| 692 | 10 | 755000 |
| 1183 | 10 | 745000 |
| 1170 | 10 | 625000 |
| 899 | 9 | 611657 |
| 804 | 9 | 582933 |
| 1047 | 9 | 556581 |
| 441 | 10 | 555000 |
| 770 | 8 | 538000 |
| 179 | 9 | 501837 |
| 799 | 9 | 485000 |
| ... | ... | ... |
| 1381 | 3 | 58500 |
| 813 | 5 | 55993 |
| 706 | 4 | 55000 |
| 1326 | 4 | 55000 |
| 1338 | 4 | 52500 |
| 711 | 3 | 52000 |
| 31 | 4 | 40000 |
| 534 | 1 | 39300 |
| 969 | 3 | 37900 |
| 917 | 2 | 35311 |
| 496 | 4 | 34900 |
</code></pre>
<h3>Grouping</h3>
<p>Grouping produces a map of column values to <em>datasets</em> with only rows with that column value:</p>
<pre><code class="language-clojure">user&gt; lucy-ds
_unnamed [3 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
| sally | 21 | opera |
user&gt; (ds/group-by-column lucy-ds :age)
{42 _unnamed [2 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| fred | 42 | pizza |
| ethel | 42 | sushi |
, 21 _unnamed [1 3]:
| :name | :age | :likes |
|-------|-----:|--------|
| sally | 21 | opera |
}
</code></pre>
<p>This structure (a map of data values to data with that value) will be familiar to even the most casual <code>clojure.core/group-by</code> enjoyer.</p>
<h3>Descriptive Stats and Group-By and DateTime Types</h3>
<p>Automatically parsing dates in a CSV dataset:</p>
<pre><code class="language-clojure">user&gt; (def stock-ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"))
#'user/stock-ds
user&gt; (ds/head stock-ds)
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [5 3]:
| symbol | date | price |
|--------|------------|------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
user&gt; (get stock-ds "date")
#tech.v3.dataset.column&lt;packed-local-date&gt;[560]
date
[2000-01-01, 2000-02-01, 2000-03-01, 2000-04-01, 2000-05-01, 2000-06-01, 2000-07-01, 2000-08-01, 2000-09-01, 2000-10-01, 2000-11-01, 2000-12-01, 2001-01-01, 2001-02-01, 2001-03-01, 2001-04-01, 2001-05-01, 2001-06-01, 2001-07-01, 2001-08-01...]
user&gt; (meta (get stock-ds "date"))
{:name "date", :datatype :packed-local-date, :n-elems 560}
</code></pre>
<p>Those dates are not strings (!)</p>
<p>So, math on them works as expected:</p>
<pre><code class="language-clojure">user&gt; (-&gt;&gt; (ds/group-by-column stock-ds "symbol")
(map (fn [[k v]] (ds/descriptive-stats v))))
(https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | MSFT | | | | MSFT | MSFT |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 15.81 | 24.74 | | 43.22 | 4.30395786E+00 | 1.16559225 | 39.81 | 28.80 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | AMZN | | | | AMZN | AMZN |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 5.970 | 47.99 | | 135.9 | 2.88913206E+01 | 0.98217538 | 64.56 | 128.8 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | IBM | | | | IBM | IBM |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 53.01 | 91.26 | | 130.3 | 1.65133647E+01 | 0.44446266 | 100.5 | 125.6 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|------------:|------------|------------|
| symbol | :string | 68 | 0 | | | GOOG | | | | GOOG | GOOG |
| date | :packed-local-date | 68 | 0 | 2004-08-01 | 2007-05-17 | | 2010-03-01 | 5.20003989E+10 | 0.00094625 | 2004-08-01 | 2010-03-01 |
| price | :float64 | 68 | 0 | 102.4 | 415.9 | | 707.0 | 1.35069851E+02 | -0.22776524 | 102.4 | 560.2 |
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv: descriptive-stats [3 12]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :mode | :max | :standard-deviation | :skew | :first | :last |
|-----------|--------------------|---------:|-----------:|------------|------------|-------|------------|--------------------:|-----------:|------------|------------|
| symbol | :string | 123 | 0 | | | AAPL | | | | AAPL | AAPL |
| date | :packed-local-date | 123 | 0 | 2000-01-01 | 2005-01-30 | | 2010-03-01 | 9.37554538E+10 | 0.00025335 | 2000-01-01 | 2010-03-01 |
| price | :float64 | 123 | 0 | 7.070 | 64.73 | | 223.0 | 6.31237823E+01 | 0.93215285 | 25.94 | 223.0 |
)
</code></pre>
<p>Note:</p>
<ul>
<li><code>group-by-column</code> returned a map of stock symbols to datasets of stocks with that symbol</li>
<li>On the next line, the function argument to <code>map</code> destructured that as tuples with each <code>v</code> a dataset</li>
<li>Finally <code>descriptive-stats</code> operates on those individually, notably doing useful min/mean/max calculations on the dates</li>
</ul>
<hr />
<div id="ElementwiseOperations"></div>
<h2>Elementwise Operations</h2>
<p>The underlying dtype-next system includes a mathematical abstraction designed to work with things like columns.</p>
<p>Using this to create new columns as lazily evaluated combinations of other columns:</p>
<pre><code class="language-clojure">user&gt; (def updated-ames
(assoc ames-ds
"TotalBath"
(dfn/+ (ames-ds "BsmtFullBath")
(dfn/* 0.5 (ames-ds "BsmtHalfBath"))
(ames-ds "FullBath")
(dfn/* 0.5 (ames-ds "HalfBath")))))
#'user/updated-ames
user&gt; (ds/head (ds/select-columns updated-ames ["BsmtFullBath" "BsmtHalfBath" "FullBath" "HalfBath" "TotalBath"]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 5]:
| BsmtFullBath | BsmtHalfBath | FullBath | HalfBath | TotalBath |
|-------------:|-------------:|---------:|---------:|----------:|
| 1 | 0 | 2 | 1 | 3.5 |
| 0 | 1 | 2 | 0 | 2.5 |
| 1 | 0 | 2 | 1 | 3.5 |
| 1 | 0 | 1 | 0 | 2.0 |
| 1 | 0 | 2 | 1 | 3.5 |
</code></pre>
<p><code>dtype-next</code>'s <code>emap</code> enables expressing arbitrary conversions from columns into columns:</p>
<pre><code class="language-clojure">user&gt; (require '[tech.v3.datatype :as dtype])
nil
user&gt; (def named-baths
(assoc updated-ames "NamedBath" (dtype/emap #(let [tbaths (double %)]
(cond
(&lt; tbaths 1.0) "almost none"
(&lt; tbaths 2.0) "somewhat doable"
(&lt; tbaths 3.0) "getting somewhere"
:else "living in style"))
:string
(updated-ames "TotalBath"))))
#'user/named-baths
user&gt; (ds/head (ds/select-columns named-baths ["TotalBath" "NamedBath"]))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [5 2]:
| TotalBath | NamedBath |
|----------:|-------------------|
| 3.5 | living in style |
| 2.5 | getting somewhere |
| 3.5 | living in style |
| 2.0 | getting somewhere |
| 3.5 | living in style |
</code></pre>
<p>Expensive houses have more bathrooms:</p>
<pre><code class="language-clojure">user&gt; (-&gt; (ds/select-columns named-baths ["TotalBath" "NamedBath" "SalePrice"])
(ds/sort-by-column "SalePrice" &gt;))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/ames-train.csv.gz [1460 3]:
| TotalBath | NamedBath | SalePrice |
|----------:|-------------------|----------:|
| 4.0 | living in style | 755000 |
| 4.5 | living in style | 745000 |
| 4.5 | living in style | 625000 |
| 3.5 | living in style | 611657 |
| 3.5 | living in style | 582933 |
| 3.5 | living in style | 556581 |
| 3.0 | living in style | 555000 |
| 4.5 | living in style | 538000 |
| 3.0 | living in style | 501837 |
| 3.5 | living in style | 485000 |
| ... | ... | ... |
| 2.0 | getting somewhere | 58500 |
| 1.0 | somewhat doable | 55993 |
| 2.0 | getting somewhere | 55000 |
| 1.0 | somewhat doable | 55000 |
| 1.0 | somewhat doable | 52500 |
| 1.0 | somewhat doable | 52000 |
| 1.0 | somewhat doable | 40000 |
| 1.0 | somewhat doable | 39300 |
| 1.0 | somewhat doable | 37900 |
| 1.0 | somewhat doable | 35311 |
| 1.0 | somewhat doable | 34900 |
</code></pre>
<h3>DateTime Types</h3>
<p>The underlying <code>dtype-next</code> system supports sophisticated <a href="https://cnuernber.github.io/dtype-next/tech.v3.datatype.datetime.html">datetime</a> processing.</p>
<p>A more real example of summarizing stock prices by year:</p>
<pre><code class="language-clojure">user&gt; (def stock-ds (ds/-&gt;dataset "https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv"
{:key-fn keyword}))
#'user/stock-ds
user&gt; (ds/head stock-ds)
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [5 3]:
| :symbol | :date | :price |
|---------|------------|-------:|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
user&gt; (meta (stock-ds :date))
{:name :date, :datatype :packed-local-date, :n-elems 560}
user&gt; (require '[tech.v3.datatype.datetime :as dtype-dt]) ;; for datetime processing
nil
user&gt; (require '[tech.v3.datatype.functional :as dfn]) ;; for functional column processing
nil
user&gt; (assoc stock-ds :year (dtype-dt/long-temporal-field :years (stock-ds :date)))
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [560 4]:
| :symbol | :date | :price | :year |
|---------|------------|-------:|------:|
| MSFT | 2000-01-01 | 39.81 | 2000 |
| MSFT | 2000-02-01 | 36.35 | 2000 |
| MSFT | 2000-03-01 | 43.22 | 2000 |
| MSFT | 2000-04-01 | 28.37 | 2000 |
| MSFT | 2000-05-01 | 25.45 | 2000 |
| MSFT | 2000-06-01 | 32.54 | 2000 |
| MSFT | 2000-07-01 | 28.40 | 2000 |
| MSFT | 2000-08-01 | 28.40 | 2000 |
| MSFT | 2000-09-01 | 24.53 | 2000 |
| MSFT | 2000-10-01 | 28.02 | 2000 |
| ... | ... | ... | ... |
| AAPL | 2009-05-01 | 135.81 | 2009 |
| AAPL | 2009-06-01 | 142.43 | 2009 |
| AAPL | 2009-07-01 | 163.39 | 2009 |
| AAPL | 2009-08-01 | 168.21 | 2009 |
| AAPL | 2009-09-01 | 185.35 | 2009 |
| AAPL | 2009-10-01 | 188.50 | 2009 |
| AAPL | 2009-11-01 | 199.91 | 2009 |
| AAPL | 2009-12-01 | 210.73 | 2009 |
| AAPL | 2010-01-01 | 192.06 | 2010 |
| AAPL | 2010-02-01 | 204.62 | 2010 |
| AAPL | 2010-03-01 | 223.02 | 2010 |
user&gt; (-&gt;&gt; (-&gt; (assoc stock-ds :year (dtype-dt/long-temporal-field :years (stock-ds :date)))
(ds/group-by (juxt :symbol :year)))
;; Now operating on the map of pairs to ds returned by `group-by`
(map (fn [[[symbol year] ds]]
{:symbol symbol
:year year
:mean-price (dfn/mean (ds :price))}))
(sort-by (juxt :symbol :year))
(ds/-&gt;&gt;dataset {:dataset-name "Mean Stock Price by Year"})
(ds/print-all))
Mean Stock Price by Year [51 3]:
| :symbol | :year | :mean-price |
|---------|------:|-------------:|
| AAPL | 2000 | 21.74833333 |
| AAPL | 2001 | 10.17583333 |
| AAPL | 2002 | 9.40833333 |
| AAPL | 2003 | 9.34750000 |
| AAPL | 2004 | 18.72333333 |
| AAPL | 2005 | 48.17166667 |
| AAPL | 2006 | 72.04333333 |
| AAPL | 2007 | 133.35333333 |
| AAPL | 2008 | 138.48083333 |
| AAPL | 2009 | 150.39333333 |
| AAPL | 2010 | 206.56666667 |
| AMZN | 2000 | 43.93083333 |
| AMZN | 2001 | 11.73916667 |
| AMZN | 2002 | 16.72333333 |
| AMZN | 2003 | 39.01666667 |
| AMZN | 2004 | 43.26750000 |
| AMZN | 2005 | 40.18750000 |
| AMZN | 2006 | 36.25166667 |
| AMZN | 2007 | 69.95250000 |
| AMZN | 2008 | 69.01500000 |
| AMZN | 2009 | 90.73083333 |
| AMZN | 2010 | 124.21000000 |
| GOOG | 2004 | 159.47600000 |
| GOOG | 2005 | 286.47250000 |
| GOOG | 2006 | 415.25666667 |
| GOOG | 2007 | 548.75833333 |
| GOOG | 2008 | 454.99916667 |
| GOOG | 2009 | 449.92000000 |
| GOOG | 2010 | 538.97666667 |
| IBM | 2000 | 96.91416667 |
| IBM | 2001 | 96.96833333 |
| IBM | 2002 | 75.12500000 |
| IBM | 2003 | 77.30750000 |
| IBM | 2004 | 83.88583333 |
| IBM | 2005 | 77.49750000 |
| IBM | 2006 | 78.71750000 |
| IBM | 2007 | 101.27666667 |
| IBM | 2008 | 107.22500000 |
| IBM | 2009 | 109.29666667 |
| IBM | 2010 | 124.85333333 |
| MSFT | 2000 | 29.67333333 |
| MSFT | 2001 | 25.34750000 |
| MSFT | 2002 | 21.82666667 |
| MSFT | 2003 | 20.93416667 |
| MSFT | 2004 | 22.67416667 |
| MSFT | 2005 | 23.84583333 |
| MSFT | 2006 | 24.75833333 |
| MSFT | 2007 | 29.28416667 |
| MSFT | 2008 | 25.20833333 |
| MSFT | 2009 | 22.87250000 |
| MSFT | 2010 | 28.50666667 |
</code></pre>
<hr />
<div id="GettingDataBackOut"></div>
<h2>Writing A Dataset Out</h2>
<p>The <code>write!</code> function puts a dataset in a file or on a stream:</p>
<pre><code class="language-clojure">(ds/write! test-ds "test.csv")
(ds/write! test-ds out-stream)
(ds/write! test-ds "test.nippy")
</code></pre>
<p>The file format is derived from the extension.</p>
<p>The excellent <a href="nippy-serialization-rocks.html">nippy</a> format is a good choice if downstream consumers are other TMD users, in which case datasets work just like any other edn data. While CSV stores numbers as ascii text, nippy stores them binary, saving some space, but the real win is 10X-100X loading performance improvement when compared to csv/tsv.</p>
<p>Moreover, because nippy knows Clojure, you can write out heterogeneous data structures that contain datasets and other things such as the result of a group-by:</p>
<pre><code class="language-clojure">user&gt; (require '[taoensso.nippy :as nippy])
nil
user&gt; (def byte-data (nippy/freeze (ds/group-by stock-ds :symbol)))
#'user/byte-data
user&gt; (type byte-data)
[B
user&gt; (keys (nippy/thaw byte-data))
("MSFT" "GOOG" "AAPL" "IBM" "AMZN")
user&gt; (first (nippy/thaw byte-data))
["MSFT"
https://github.com/techascent/tech.ml.dataset/raw/master/test/data/stocks.csv [123 3]:
| :symbol | :date | :price |
|---------|------------|--------|
| MSFT | 2000-01-01 | 39.81 |
| MSFT | 2000-02-01 | 36.35 |
| MSFT | 2000-03-01 | 43.22 |
| MSFT | 2000-04-01 | 28.37 |
| MSFT | 2000-05-01 | 25.45 |
</code></pre>
<p>For Apache arrow support, see the <a href="https://techascent.github.io/tech.ml.dataset/tech.v3.libs.arrow.html">tech.v3.libs.arrow</a> documentation.</p>
<p>For Parquet, see the <a href="https://techascent.github.io/tech.ml.dataset/tech.v3.libs.parquet.html">tech.v3.libs.parquet</a> documentation.</p>
<hr />
<h2>Further Reading</h2>
<ul>
<li>
<p>The <a href="https://techascent.github.io/tech.ml.dataset/200-quick-reference.html">quick reference</a> summarizes many of the most frequently used functions with hints about their use.</p>
</li>
<li>
<p>The <a href="https://techascent.github.io/tech.ml.dataset/index.html">API docs</a> list every function available in TMD.</p>
</li>
</ul>
</div></div></div></body></html>
Reference in New Issue View Git Blame Copy Permalink