atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Ibis Project Blog</title><link href="http://www.ibis-project.org/" rel="alternate"></link><link href="/atom.xml" rel="self"></link><id>http://www.ibis-project.org/</id><updated>2016-06-14T10:00:00-07:00</updated><entry><title>Announcing hs2client, a fast new C++ / Python Thrift client for Impala and Hive</title><link href="http://www.ibis-project.org/new-native-impala-hive-client/" rel="alternate"></link><updated>2016-06-14T10:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2016-06-14:new-native-impala-hive-client/</id><summary type="html">

&lt;p&gt;This year, I collaborated with members of the &lt;a href="http://impala.io/"&gt;Apache Impala (incubating)&lt;/a&gt;
team at Cloudera to create a new C++ library to eventually become a faster,
more memory-efficient replacement for &lt;a href="https://github.com/cloudera/impyla"&gt;impyla&lt;/a&gt;, &lt;a href="https://github.com/dropbox/PyHive"&gt;PyHive&lt;/a&gt;, and other
(largely pure Python) client libraries for talking to Apache Hive and Impala.&lt;/p&gt;
&lt;p&gt;We are excited to release this effort, dubbed &lt;strong&gt;hs2client&lt;/strong&gt;, as a new
Apache-licensed &lt;a href="https://github.com/cloudera/hs2client"&gt;open source project on GitHub&lt;/a&gt;. As you may guess from the
name, this library implement the HiveServer2 Thrift API as a C++ library, with
careful handling of result sets to allow languages like Python to access data
at high performance.&lt;/p&gt;
&lt;p&gt;The initial preview release contains:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;A C++ library, &lt;code&gt;libhs2client&lt;/code&gt;, which provides a clean C++ API for the
  HiveServer2 Thrift API. This can be built and dynamically or statically
  linked in C/C++ applications with no direct exposure to Apache Thrift.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Python bindings, with optimized reads to &lt;strong&gt;pandas.DataFrame&lt;/strong&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To try out the library, you can install a dev build of the project right now
with conda:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;conda install hs2client -c cloudera/channel/dev
&lt;/pre&gt;&lt;/div&gt;




&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="optimizing-for-speed-and-memory-use"&gt;Optimizing for speed and memory use&lt;/h2&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;The Python bindings for &lt;code&gt;libhs2client&lt;/code&gt; are careful about performance and memory in a few key ways:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Converting row batches directly into pandas-compatible NumPy arrays&lt;/li&gt;
&lt;li&gt;Care with categorical data: string data is deduplicated while it's being converted for pandas. In the future, it would be straightforward to add an option to return all string data as &lt;code&gt;pandas.Categorical&lt;/code&gt; arrays&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="hs2client-demo-and-simple-benchmark"&gt;hs2client demo and simple benchmark&lt;/h2&gt;
&lt;p&gt;The initial API is oriented at modeling the HiveServer2 protocol closely. Connecting to a cluster yields a &lt;code&gt;Service&lt;/code&gt; instance:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;hs2client&lt;/span&gt;

&lt;span class="n"&gt;service&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hs2client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;localhost&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;21050&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;wesm&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To execute queries, you open a session then use its &lt;code&gt;execute&lt;/code&gt; method.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open_session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select 1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Queries are asynchronous by default.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_finished&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[3]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
True
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_state&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
&amp;apos;finished&amp;apos;
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Result sets can be fetch currently to &lt;code&gt;pandas.DataFrame&lt;/code&gt; objects:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchall_pandas&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[6]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[6]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;1&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;1&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Compared with comparable database clients, we've optimized the fetch path to have excellent performance (IO speeds) and memory use.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[10]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;numpy&lt;/span&gt; &lt;span class="kn"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;np&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;pandas&lt;/span&gt; &lt;span class="kn"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;pd&lt;/span&gt;

&lt;span class="n"&gt;N&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;

&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;randn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;N&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                  &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;data{0}&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;localhost&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                          &lt;span class="n"&gt;hdfs_client&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hdfs_connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5070&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;drop_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;hs2_test&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;force&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;hs2_test&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;test_data&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;hs2_test&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[16]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute_sync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;select * from hs2_test.test_data&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="k"&gt;time&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchall_pandas&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
CPU times: user 200 ms, sys: 64 ms, total: 264 ms
Wall time: 682 ms

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[21]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;speed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory_usage&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;0.682&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Speed: {0:.2f} MB/s&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;speed&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
Speed: 111.87 MB/s

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This benchmark is being run on &lt;code&gt;localhost&lt;/code&gt;, but it shows we can move over 110 MB/s through the Thrift protocol &lt;strong&gt;and&lt;/strong&gt; convert to a fully-formed pandas DataFrame.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="hs2client-compared-with-impyla"&gt;hs2client compared with impyla&lt;/h2&gt;
&lt;p&gt;In Ibis and other Python projects, we have been using &lt;a href="http://github.com/cloudera/impyla"&gt;impyla&lt;/a&gt; to execute queries and access result sets. impyla either uses Apache Thrift's official Python implementation (on Python 2) or &lt;a href="https://github.com/eleme/thriftpy"&gt;thriftpy&lt;/a&gt; (on Python 3) for interacting with the Impala or Hive Thrift service. Because of this, the main difference is the performance in fetching large result sets.&lt;/p&gt;
&lt;p&gt;Here, I will perform an equivalent fetch using impyla via ibis:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[29]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# Connect to Impala via impyla (on Python 2)&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;localhost&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;21050&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;select * from hs2_test.test_data&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="k"&gt;time&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
CPU times: user 1.15 s, sys: 52 ms, total: 1.2 s
Wall time: 1.66 s

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[30]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;speed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory_usage&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1.66&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Speed: {0:.2f} MB/s&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;speed&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
Speed: 45.96 MB/s

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;In this particular use case with an all-numeric table (which is ideal for deserialization performance), the speed difference is only 2x or a bit more.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="hs2client-roadmap"&gt;hs2client roadmap&lt;/h2&gt;
&lt;p&gt;The &lt;a href="http://github.com/cloudera/hs2client"&gt;code is hosted on GitHub&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The C++ library does not yet implement some important features that are needed to be a drop-in replacement for impyla or other Hive or Impala drivers:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;SSL transport (with certificate verification)&lt;/li&gt;
&lt;li&gt;SASL Thrift transport for secure (i.e. Kerberos) clusters, or insecure clusters using SASL&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On the Python side, we must implement a DB API 2.0 compatibility layer, since currently data can only be fetched to pandas, not Python tuples as with most Python database drivers.&lt;/p&gt;
&lt;p&gt;We of course welcome contributions from the community to build out some of these features.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Ibis 0.8: Initial PostgreSQL support, bug fixes</title><link href="http://www.ibis-project.org/release-0.8/" rel="alternate"></link><updated>2016-05-19T08:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2016-05-19:release-0.8/</id><summary type="html">

&lt;p&gt;Thanks to contributions from pandas core team member &lt;a href="http://github.com/cpcloud"&gt;Philip Cloud&lt;/a&gt;, Ibis
now has initial support for PostgreSQL. The 0.8 release also includes many bug
fixes from 0.7, and all Ibis users are recommended to upgrade.&lt;/p&gt;
&lt;p&gt;Read more in the &lt;a href="http://docs.ibis-project.org/release.html"&gt;release notes&lt;/a&gt;.&lt;/p&gt;


&lt;h2&gt;What is new with Ibis&lt;/h2&gt;
&lt;p&gt;Ibis development has been slower in 2016 as I've been investing energies in
&lt;a href="http://arrow.apache.org"&gt;Apache Arrow&lt;/a&gt;, &lt;a href="http://parquet.apache.org"&gt;Apache Parquet&lt;/a&gt;, and other projects that are all part of
the broader goal of making Python work better with Hadoop and other distributed
data systems. Some of the this functionality, like the ability to read Parquet
files natively in Python, will start showing up in Ibis in the relatively near
future.&lt;/p&gt;
&lt;p&gt;In the meantime, we've been stabilizing and improving the existing Ibis
functionality while working with the community to bring about new features.&lt;/p&gt;
&lt;h2&gt;PostgreSQL support in Ibis&lt;/h2&gt;
&lt;p&gt;The PostgreSQL database contains vast analytical functionality, far too much to
cover completely right away, but Ibis now has support for a meaningful and
useful subset of Postgres's built-in functions. It is also capable of
performing window functions as well as a many date, string, and mathematical
operations. If you find a Postgres function you'd like to see made available in
Ibis, please let us know on the GitHub issue tracker.&lt;/p&gt;
&lt;p&gt;Here's an example of what connecting to a PostgreSQL database and executing an
expression looks like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;postgres&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;localhost&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ibis_test&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                                       &lt;span class="n"&gt;password&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ibis_test&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                                       &lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ibis_testing&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list_tables&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;Out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;u&amp;#39;functional_alltypes&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;functional_alltypes&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;string_col&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;Out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
  &lt;span class="n"&gt;string_col&lt;/span&gt;  &lt;span class="n"&gt;count&lt;/span&gt;
&lt;span class="mi"&gt;0&lt;/span&gt;          &lt;span class="mi"&gt;8&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;1&lt;/span&gt;          &lt;span class="mi"&gt;9&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;2&lt;/span&gt;          &lt;span class="mi"&gt;2&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;3&lt;/span&gt;          &lt;span class="mi"&gt;5&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;4&lt;/span&gt;          &lt;span class="mi"&gt;3&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;5&lt;/span&gt;          &lt;span class="mi"&gt;1&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;6&lt;/span&gt;          &lt;span class="mi"&gt;7&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;7&lt;/span&gt;          &lt;span class="mi"&gt;0&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;8&lt;/span&gt;          &lt;span class="mi"&gt;4&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;span class="mi"&gt;9&lt;/span&gt;          &lt;span class="mi"&gt;6&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;</summary></entry><entry><title>Ibis 0.7: Kudu-Impala integration, SQL compiler improvements</title><link href="http://www.ibis-project.org/release-0.7/" rel="alternate"></link><updated>2016-03-16T05:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2016-03-16:release-0.7/</id><summary type="html">

&lt;p&gt;Ibis 0.7.0 has been released! The biggest new feature in the release is
&lt;a href="http://blog.ibis-project.org/kudu-impala-ibis/"&gt;Impala-Kudu integration&lt;/a&gt;. This is great timing, because &lt;a href="https://github.com/apache/incubator-kudu/tree/master/python"&gt;Kudu's Python client&lt;/a&gt; went beta officially in its recent 0.7.0 release.&lt;/p&gt;


&lt;p&gt;In addition to many bug fixes, Ibis includes a much smarter SQL compiler for
more complex pandas-like expressions. For example, consider the following
operation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;([(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;flag&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;string&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;value&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;double&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt;
                   &lt;span class="s1"&gt;&amp;#39;tbl&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;flagged&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;1&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;unflagged&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;fv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;flagged&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;
&lt;span class="n"&gt;uv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;unflagged&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;fv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;uv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;uv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Now in Ibis 0.7.0, this expression can be transformed to the correct effective
SQL:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;avg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;tbl&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;1&amp;#39;&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;
  &lt;span class="k"&gt;CROSS&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;avg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;tbl&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;0&amp;#39;&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;t1&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Thanks to all who contributed patches:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$ git log v0.6.0..v0.7.0 --pretty&lt;span class="o"&gt;=&lt;/span&gt;format:%aN &lt;span class="p"&gt;|&lt;/span&gt; sort &lt;span class="p"&gt;|&lt;/span&gt; uniq -c &lt;span class="p"&gt;|&lt;/span&gt; sort -rn
    &lt;span class="m"&gt;21&lt;/span&gt; Wes McKinney
     &lt;span class="m"&gt;1&lt;/span&gt; Uri Laserson
     &lt;span class="m"&gt;1&lt;/span&gt; Kristopher Overholt
&lt;/pre&gt;&lt;/div&gt;</summary></entry><entry><title>Interactive Analytics on Dynamic Big Data in Python using Kudu, Impala, and Ibis</title><link href="http://www.ibis-project.org/kudu-impala-ibis/" rel="alternate"></link><updated>2015-12-15T08:00:00-08:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-12-15:kudu-impala-ibis/</id><summary type="html">

&lt;p&gt;The new &lt;a href="http://getkudu.io"&gt;Apache Kudu (incubating)&lt;/a&gt; columnar storage engine together with
&lt;a href="http://impala.io"&gt;Apache Impala (incubating)&lt;/a&gt; interactive SQL engine enable a new fully open
source big data architecture for data that is arriving and changing very
quickly. By integrating Kudu and Impala with Ibis, this functionality is now
available to Python programmers with an easy-to-use pandas-like API.&lt;/p&gt;
&lt;p&gt;I spent this last week expanding the Kudu Python client (a Cython wrapper for
the C++ client API) and adding initial integration with Ibis. While my &lt;a href="http://gerrit.cloudera.org:8080/#/c/1593/"&gt;Kudu patch&lt;/a&gt; is still in code review, I will give you a preview here of how it all
works.&lt;/p&gt;
&lt;p&gt;Since Kudu, a native C++ storage engine, now builds on OS X, I'm writing this
blog using the Kudu client on OS X, so now is a great time for developers on
both OS X and Linux to get involved with developing Python tools for Kudu.&lt;/p&gt;
&lt;p&gt;Using Impala on OS X is in the works, see the &lt;a href="https://issues.cloudera.org/browse/IMPALA-2761"&gt;issue tracker&lt;/a&gt; for more. As
soon as we can, we'll provide a combined Kudu/Impala DMG installer for
installing the environment locally on Mac computers.&lt;/p&gt;
&lt;!--
If you have XCode installed, you can follow these steps to get going:

wzxhzdk:0

--&gt;

&lt;p&gt;Read on for more about Kudu and a full stack demo.&lt;/p&gt;


&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="kudu-overview"&gt;Kudu overview&lt;/h2&gt;
&lt;p&gt;Kudu was designed for the Hadoop ecosystem in part to simplify architectures involving very fast-arriving and fast-changing data that needs to be immediately available for analytical queries. In the past, complex architectures were devised using the fast &lt;a href="https://parquet.apache.org/"&gt;Parquet&lt;/a&gt; columnar format stored in HDFS in conjunction with &lt;a href="https://hbase.apache.org/"&gt;HBase&lt;/a&gt; (for new data, but very slow for analytics), but there were numerous drawbacks that made a purpose-built column-oriented storage engine desirable. For example, while Parquet is extremely fast for analytics, data can only be appended to a dataset and not deleted or updated.&lt;/p&gt;
&lt;p&gt;You can read much more about Kudu in Todd Lipcon's &lt;a href="http://www.slideshare.net/cloudera/kudu-new-hadoop-storage-for-fast-analytics-on-fast-data"&gt;recent slide deck&lt;/a&gt; and in an &lt;a href="http://getkudu.io/overview.html"&gt;website overview&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;ZoomData put together a &lt;a href="https://www.youtube.com/watch?v=ck_kRb6qLKE"&gt;cool demo&lt;/a&gt; showing a real time analytics dashboard powered by Impala and Kudu.&lt;/p&gt;
&lt;p&gt;For Python users, here are the key details:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Kudu has a SQL-like tabular data model; table columns are typed, and columns can be added and remove from tables. A Kudu cluster can store any number of tables.&lt;/li&gt;
&lt;li&gt;Tables must have one or more &lt;em&gt;primary keys&lt;/em&gt;. Like SQL databases, these will impact the performance of retrieving individual records.&lt;/li&gt;
&lt;li&gt;Data is stored column-oriented, and individual table columns can be read (or &lt;em&gt;scanned&lt;/em&gt;) very fast.&lt;/li&gt;
&lt;li&gt;You can mutate a table by adding, deleting, or updating rows.&lt;/li&gt;
&lt;li&gt;Data can be selected by indicating a number of conditions or &lt;em&gt;predicates&lt;/em&gt; that must hold true&lt;/li&gt;
&lt;li&gt;Kudu does not perform analytics: its job is to manage tabular data and serve it to compute engines as fast as possible.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Kudu is not coupled to any particular compute engine. Any system that can use its C++ or Java clients can use it. There are a few compute system integrations built or in progress:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="http://getkudu.io/docs/kudu_impala_integration.html"&gt;&lt;strong&gt;Apache Impala (incubating)&lt;/strong&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://issues.cloudera.org/browse/KUDU-1214"&gt;&lt;strong&gt;Apache Spark&lt;/strong&gt;&lt;/a&gt;: Still in active development&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/dremio/drill-storage-kudu"&gt;&lt;strong&gt;Apache Drill&lt;/strong&gt;&lt;/a&gt;: Built during the recent Drill-Kudu hackathon&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="basic-kudu-use-in-python"&gt;Basic Kudu use in Python&lt;/h2&gt;
&lt;p&gt;Let's take our first steps in Python. I've booted up the Kudu Quickstart VM (for VirtualBox) that you can download from the Kudu website. I've installed the Kudu Python client and now import it and connect to the Kudu master in the VM:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;kudu&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;quickstart.cloudera&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;7051&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Since this is a brand new cluster, there are no tables created yet:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list_tables&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[2]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To create one, we first create a schema and then create the table:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema_builder&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_column&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;id&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nullable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_column&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;item&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_column&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;price&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;double&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_primary_keys&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;id&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;build&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;schema&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[3]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
kudu.Schema {  
  id     int64(nullable=False) PRIMARY KEY
  item   string(nullable=True)
  price  double(nullable=True)
}
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table_exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;purchases&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;delete_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;purchases&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;purchases&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list_tables&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[&amp;apos;purchases&amp;apos;]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, we can get a handle for this new table and see its schema:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;purchases&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;purchases&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[5]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
&amp;lt;kudu.client.Table at 0x1037fc8f0&amp;gt;
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[6]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[6]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
kudu.Schema {  
  id     int64(nullable=False) PRIMARY KEY
  item   string(nullable=True)
  price  double(nullable=True)
}
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, let's insert some data:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[7]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;spam&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.49&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;eggs&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.25&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;coffee&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.35&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;spam&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.00&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;eggs&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.49&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;coffee&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.75&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;eggs&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.75&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;coffee&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.95&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;spam&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;3.00&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;eggs&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.25&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;eggs&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.00&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;coffee&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.35&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To add, change, or remove data from a table, you must create a &lt;em&gt;session&lt;/em&gt; to group the operations:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[8]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;new_session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, you create insert operations and add them to the session and call its &lt;code&gt;flush&lt;/code&gt; method:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[9]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_price&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;new_insert&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;id&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_id&lt;/span&gt;
    &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;item&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_item&lt;/span&gt;
    &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;price&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_price&lt;/span&gt;
    &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;apply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    
&lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, suppose we wanted to select some data from the table. To do this, we create a &lt;em&gt;scanner&lt;/em&gt; for the table in question:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[10]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To read all of the data out, you &lt;em&gt;open&lt;/em&gt; the scanner and call one of its read methods:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[11]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read_all_tuples&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[11]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[(1, &amp;apos;spam&amp;apos;, 2.49),
 (2, &amp;apos;eggs&amp;apos;, 1.25),
 (3, &amp;apos;coffee&amp;apos;, 2.35),
 (4, &amp;apos;spam&amp;apos;, 2.0),
 (5, &amp;apos;eggs&amp;apos;, 2.49),
 (6, &amp;apos;coffee&amp;apos;, 2.75),
 (7, &amp;apos;eggs&amp;apos;, 2.75),
 (8, &amp;apos;coffee&amp;apos;, 1.95),
 (9, &amp;apos;spam&amp;apos;, 3.0),
 (10, &amp;apos;eggs&amp;apos;, 2.25),
 (11, &amp;apos;eggs&amp;apos;, 2.0),
 (12, &amp;apos;coffee&amp;apos;, 2.35)]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To only read a particular subset of data, you add &lt;em&gt;predicates&lt;/em&gt; to the scanner:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[12]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_predicate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;item&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;spam&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read_all_tuples&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[12]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[(1, &amp;apos;spam&amp;apos;, 2.49), (4, &amp;apos;spam&amp;apos;, 2.0), (9, &amp;apos;spam&amp;apos;, 3.0)]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[13]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;id_col&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;id&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_predicates&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;id_col&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id_col&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read_all_tuples&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[13]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[(5, &amp;apos;eggs&amp;apos;, 2.49),
 (6, &amp;apos;coffee&amp;apos;, 2.75),
 (7, &amp;apos;eggs&amp;apos;, 2.75),
 (8, &amp;apos;coffee&amp;apos;, 1.95),
 (9, &amp;apos;spam&amp;apos;, 3.0),
 (10, &amp;apos;eggs&amp;apos;, 2.25)]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;That's all we need to know for now. There's lots more to know about Kudu and things that can be added to the Python interface, such as:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Native pandas DataFrame read/write capability&lt;/li&gt;
&lt;li&gt;Hash and range partition configuration&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="querying-existing-kudu-tables-with-ibis-and-impala"&gt;Querying existing Kudu tables with Ibis and Impala&lt;/h2&gt;
&lt;p&gt;In the latest development version of Ibis, you can add Kudu to the mix when working with Impala. Let's take a look:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[14]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;
&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactive&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;

&lt;span class="n"&gt;host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;quickstart.cloudera&amp;#39;&lt;/span&gt;

&lt;span class="n"&gt;hdfs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hdfs_connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;50070&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;ic&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;21050&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hdfs_client&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hdfs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This Impala cluster is built with Kudu support, so I can connect my Ibis client to the Kudu master like so:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[15]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;7051&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, let's see about that data we just wrote:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[16]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list_tables&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[16]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[&amp;apos;purchases&amp;apos;]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;The &lt;code&gt;table&lt;/code&gt; method on &lt;code&gt;ic.kudu&lt;/code&gt; automatically creates an Impala table whose metadata references the existing data in Kudu:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[17]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;purchases&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[18]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[18]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
    id    item  price
0    1    spam   2.49
1    2    eggs   1.25
2    3  coffee   2.35
3    4    spam   2.00
4    5    eggs   2.49
5    6  coffee   2.75
6    7    eggs   2.75
7    8  coffee   1.95
8    9    spam   3.00
9   10    eggs   2.25
10  11    eggs   2.00
11  12  coffee   2.35
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;The result behaves just like any other Ibis table, such as those you might have used with HDFS or SQLite:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[19]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;spam&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[19]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   id  item  price
0   1  spam   2.49
1   4  spam   2.00
2   9  spam   3.00
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[20]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;item&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[20]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
     item  mean(price)
0    spam     2.496667
1  coffee     2.350000
2    eggs     2.148000
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;You can issue &lt;code&gt;SELECT&lt;/code&gt;, &lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;DELETE&lt;/code&gt;, and &lt;code&gt;UPDATE&lt;/code&gt; queries on data in Kudu tables via Impala, but for now only &lt;code&gt;SELECT&lt;/code&gt; and &lt;code&gt;INSERT&lt;/code&gt; operations are available from Ibis.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="creating-new-kudu-backed-impala-tables"&gt;Creating new Kudu-backed Impala tables&lt;/h2&gt;
&lt;p&gt;The Impala client's Kudu interface has a method &lt;code&gt;create_table&lt;/code&gt; which enables more flexible Impala table creation with data stored in Kudu. This includes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Creating empty tables with a particular schema&lt;/li&gt;
&lt;li&gt;Creating tables from an Ibis table expression (i.e. a &amp;quot;CTAS&amp;quot; in database speak)&lt;/li&gt;
&lt;li&gt;Creating tables from pandas DataFrame objects&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Let's look at a couple examples.&lt;/p&gt;
&lt;p&gt;First, I'll create an empty Kudu-backed table:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[21]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;([(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;foo&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;int32&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                      &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bar&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;string&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                      &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;baz&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;double&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exists_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;kudu_tables&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;drop_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;kudu_tables&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;force&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;kudu_tables&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;impala_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;example1&amp;#39;&lt;/span&gt;
&lt;span class="n"&gt;kudu_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;example1-kudu-table&amp;#39;&lt;/span&gt;
&lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;impala_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kudu_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                     &lt;span class="n"&gt;primary_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;foo&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                     &lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;kudu_tables&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;db&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;kudu_tables&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[22]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;impala_table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;impala_name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;impala_table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[22]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.Schema {  
  foo  int32
  bar  string
  baz  double
}
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's look in Kudu!&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[23]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list_tables&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[23]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[&amp;apos;purchases&amp;apos;, &amp;apos;example1-kudu-table&amp;apos;]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[24]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;example1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kudu_name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[25]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;example1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[25]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
kudu.Schema {  
  foo  int32(nullable=False) PRIMARY KEY
  bar  string(nullable=True)
  baz  double(nullable=True)
}
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Note that Impala has neither the notion of primary keys nor non-nullable fields, but this metadata can inform query planning.&lt;/p&gt;
&lt;p&gt;Now, I'll create a table by inserting a subset of the &lt;code&gt;purchases&lt;/code&gt; table:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[26]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;purchases&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;spam&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kudu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;example2&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;example2-kudu&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                     &lt;span class="n"&gt;primary_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;id&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                     &lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;kudu_tables&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[27]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example2&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[27]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   id  item  price
0   1  spam   2.49
1   4  spam   2.00
2   9  spam   3.00
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;We can look directly in Kudu to see the data that was inserted.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[28]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;example2_data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;example2-kudu&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;example2_data&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read_all_tuples&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[28]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[(1, &amp;apos;spam&amp;apos;, 2.49), (4, &amp;apos;spam&amp;apos;, 2.0), (9, &amp;apos;spam&amp;apos;, 3.0)]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="in-the-context-of-the-great-decoupling"&gt;In the context of the &amp;quot;Great Decoupling&amp;quot;&lt;/h2&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Kudu's &amp;quot;bring your own SQL&amp;quot; design is well in line with the broader decoupling and commoditization of open source storage and compute systems that has been going on the last 10 years. It is the responsibility of productivity-centric programming interfaces like Ibis (which you can think of as a &amp;quot;UI for developers&amp;quot;) to enhance interoperability and hide as much complexity from the user as possible.&lt;/p&gt;
&lt;center&gt;
&lt;img src="../images/decoupling-kudu-ibis-impala.png" width="50%"&gt;&lt;/img&gt;
&lt;/center&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="conclusions"&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;Kudu is an exciting new open source storage technology which, when combined with a high performance compute engine like Impala, enables scalable high performance analytics on fast-changing data sets. Having this functionality seamlessly available to Python programmers using Ibis will make it much easier to develop end-to-end applications involving big data analytics. Most importantly, the code that you write will be largely the same whether you have 1000 or 100 billion rows of data.&lt;/p&gt;
&lt;p&gt;I've been working to build out the Kudu Python interface so that it's easier for Python users to use the project and participate in the development community. If you see something here that interests you, we'd love to have you involved.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Ibis 0.6: Expanded Impala support, better pandas integration</title><link href="http://www.ibis-project.org/release-0.6/" rel="alternate"></link><updated>2015-12-01T05:00:00-08:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-12-01:release-0.6/</id><summary type="html">

&lt;p&gt;Ibis 0.6.0 has been released! This release brings substantial new functionality
for Impala users; see the &lt;a href="http://docs.ibis-project.org/impala.html" target="_blank"&gt;Using Ibis with Impala guide&lt;/a&gt;
for a deep dive on these features. These enable Ibis to solve many data
management problems beyond the analytical SQL capabilities exposed through Ibis
expressions.&lt;/p&gt;
&lt;p&gt;Some highlights of the new functionality include:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Create tables and insert into existing tables directly from pandas DataFrame
  objects&lt;/li&gt;
&lt;li&gt;Inspect a variety of table and partition metadata in a user-friendly manner&lt;/li&gt;
&lt;li&gt;Comprehensive partitioned table support&lt;/li&gt;
&lt;li&gt;Modify table and partition metadata&lt;/li&gt;
&lt;li&gt;Set session options such as Parquet compression codec&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Several bugs were fixed as well.&lt;/p&gt;
&lt;p&gt;The upcoming roadmap will feature Impala's nested types support and expanded
SQL system coverage.&lt;/p&gt;


&lt;p&gt;Thanks to all who contributed patches:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$ git log v0.5.0..v0.6.0 --pretty&lt;span class="o"&gt;=&lt;/span&gt;format:%aN &lt;span class="p"&gt;|&lt;/span&gt; sort &lt;span class="p"&gt;|&lt;/span&gt; uniq -c &lt;span class="p"&gt;|&lt;/span&gt; sort -rn
  &lt;span class="m"&gt;46&lt;/span&gt; Wes McKinney
   &lt;span class="m"&gt;3&lt;/span&gt; Uri Laserson
   &lt;span class="m"&gt;1&lt;/span&gt; Phillip Cloud
   &lt;span class="m"&gt;1&lt;/span&gt; mariusvniekerk
   &lt;span class="m"&gt;1&lt;/span&gt; Kristopher Overholt
&lt;/pre&gt;&lt;/div&gt;</summary></entry><entry><title>Ibis for SQL Programmers Guide</title><link href="http://www.ibis-project.org/ibis-for-sql-programmers/" rel="alternate"></link><updated>2015-09-17T10:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-09-17:ibis-for-sql-programmers/</id><summary type="html">

&lt;p&gt;Yesterday, on the Ibis documentation I published the first version of &lt;a href="http://docs.ibis-project.org/sql.html" target="_blank"&gt;Ibis for SQL Programmers&lt;/a&gt;, a practical guide to how SQL concepts
map onto the Ibis DSL. To be clear, all of your data will stay in a SQL engine,
but you can interact with it using Ibis using Python code that can be simpler
to write and also more composable and reusable. Also, Ibis statically analyzes
types and expressions as you go, catching a large class of common mistakes made
during SQL programming.&lt;/p&gt;
&lt;p&gt;This will be an evolving document, so if you find something that you use
commonly in SQL and would like to see it added, please &lt;a href="https://github.com/cloudera/ibis/issues" target="_blank"&gt;reach out to us&lt;/a&gt; on the project issue tracker.&lt;/p&gt;
</summary></entry><entry><title>Ibis Design: Expression types and operation mechanics</title><link href="http://www.ibis-project.org/design-types-operations/" rel="alternate"></link><updated>2015-09-13T10:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-09-13:design-types-operations/</id><summary type="html">&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;


&lt;p&gt;This blog post describes Ibis's expression type system and how its DSL (domain-specific language) operations work. It's intended for curious users or developers of the project.&lt;/p&gt;
&lt;p&gt;I spent a lot of time thinking about how to create a deferred expression DSL that compiles to SQL and is fluid and intuitive to use. No small task.&lt;/p&gt;
&lt;p&gt;The key things I want to address are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Ibis's internal operation types and external user-facing types&lt;/li&gt;
&lt;li&gt;How the DSL semantics help you avoid making common mistakes&lt;/li&gt;
&lt;li&gt;How type checking and evaluation works&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;At the end, since several people have asked me about it, I show some contrasts in Ibis's API design with &lt;a href="http://blaze.pydata.org/"&gt;Blaze&lt;/a&gt;, another Python project, that has its own embedded DSL.&lt;/p&gt;


&lt;h2 id="ibis-expression-basics"&gt;Ibis expression basics&lt;/h2&gt;
&lt;p&gt;In general, all objects in Ibis semantically referencing real data fall into 3 categories:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Tables: Database tables, data in formats like Parquet and Avro, or derived table expressions&lt;/li&gt;
&lt;li&gt;Arrays: Columns in tables or derived expressions thereof&lt;/li&gt;
&lt;li&gt;Scalars: Literal values and results of aggregations or analytic functions&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I'll give you a bunch of basic examples of these and then give more technical detail on the type system.&lt;/p&gt;
&lt;p&gt;Let's just take the number 5:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;

&lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Suppose you wanted to create an expression that uses this value. Ibis has the function &lt;code&gt;literal&lt;/code&gt; that converts a Python scalar value to a well-typed expression. Most of the time this happens automatically, but let's do it explicitly:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;int_expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;literal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;int_expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[2]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
Literal[int8]
  5
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This code didn't actually &lt;em&gt;do&lt;/em&gt; anything. It created a &lt;em&gt;literal expression&lt;/em&gt; containing the value 5. What you see in brackets is the output type of the operation &lt;code&gt;Literal&lt;/code&gt;. What is &lt;code&gt;int_expr&lt;/code&gt;, then?&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;int_expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[3]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.Int8Scalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Okay, let's figure out this &lt;code&gt;int_expr&lt;/code&gt; thing.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;int_expr&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
Add[int8]
  Literal[int8]
    5
  Literal[int8]
    10
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;int_expr&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[5]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.Int8Scalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[6]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;int_expr&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[6]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
Sqrt[double]
  Add[int8]
    Literal[int8]
      5
    Literal[int8]
      10
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Read the expression tree inside out. So &lt;code&gt;10&lt;/code&gt; got automatically converted to a literal, then they're &lt;code&gt;Add&lt;/code&gt;-ed together, finally &lt;code&gt;Sqrt&lt;/code&gt; is applied. Note that:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[7]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[7]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.DoubleScalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's round it to 2 decimal places for good measure:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[8]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;rounded&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Type is {0!r}&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rounded&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;

&lt;span class="n"&gt;rounded&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
Type is &amp;lt;class &amp;apos;ibis.expr.types.DoubleScalar&amp;apos;&amp;gt;

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[8]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
Round[double]
  Sqrt[double]
    Add[int8]
      Literal[int8]
        5
      Literal[int8]
        10
  Literal[int8]
    2
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, if you wanted to run this expression inside Impala, it would get translated to this SQL:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[9]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rounded&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
SELECT round(sqrt(5 + 10), 2) AS &amp;#96;tmp&amp;#96;

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Indeed:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[10]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bottou01.sjc.cloudera.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rounded&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[10]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
3.8700000000000001
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's look at a table now:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[11]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ibis_testing&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;functional_alltypes&lt;/span&gt;
&lt;span class="n"&gt;t&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[11]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
DatabaseTable[table]
  name: ibis_testing.&amp;#96;functional_alltypes&amp;#96;
  schema:
    id : int32
    bool_col : boolean
    tinyint_col : int8
    smallint_col : int16
    int_col : int32
    bigint_col : int64
    float_col : float
    double_col : double
    date_string_col : string
    string_col : string
    timestamp_col : timestamp
    year : int32
    month : int32
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[12]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[12]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.impala.client.ImpalaTable
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This says &lt;code&gt;ImpalaTable&lt;/code&gt;, but it's really &lt;code&gt;TableExpr&lt;/code&gt; with a few extra methods that are Impala-specific. For experienced Pythonistas, you'll know the &lt;code&gt;mro&lt;/code&gt; method gives you the full inheritance hierarchy:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[13]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mro&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[13]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[ibis.impala.client.ImpalaTable,
 ibis.expr.types.TableExpr,
 ibis.expr.types.Expr,
 ibis.client.DatabaseEntity,
 object]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now let's compute the same expression from above on &lt;code&gt;tinyint_col&lt;/code&gt;, and sum it for kicks. I'll write it with one line per operation and add comments that show the operation name and output type of each operation:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[14]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;             &lt;span class="c1"&gt;# DatabaseTable -&amp;gt; TableExpr&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;    &lt;span class="c1"&gt;# Limit         -&amp;gt; TableExpr&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tinyint_col&lt;/span&gt;  &lt;span class="c1"&gt;# TableColumn   -&amp;gt; Int8Array&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;      &lt;span class="c1"&gt;# Add           -&amp;gt; Int16Array&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;       &lt;span class="c1"&gt;# Sqrt          -&amp;gt; DoubleArray&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;     &lt;span class="c1"&gt;# Round         -&amp;gt; DoubleArray&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;       &lt;span class="c1"&gt;# Sum           -&amp;gt; DoubleScalar&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[14]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ref_0
DatabaseTable[table]
  name: ibis_testing.&amp;#96;functional_alltypes&amp;#96;
  schema:
    id : int32
    bool_col : boolean
    tinyint_col : int8
    smallint_col : int16
    int_col : int32
    bigint_col : int64
    float_col : float
    double_col : double
    date_string_col : string
    string_col : string
    timestamp_col : timestamp
    year : int32
    month : int32

sum = Sum[double]
  Round[array(double)]
    Sqrt[array(double)]
      Add[array(int16)]
        tinyint_col = Column[array(int8)] &amp;apos;tinyint_col&amp;apos; from table 
          Limit[table]
            Table: ref_0
            n:
              20
            offset:
              0
        Literal[int8]
          10
    Literal[int8]
      2
  None
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;As desired:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[15]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[15]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.DoubleScalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;And:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[16]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[16]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
75.760000000000005
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[17]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
SELECT sum(round(sqrt(&amp;#96;tinyint_col&amp;#96; + 10), 2)) AS &amp;#96;sum&amp;#96;
FROM (
  SELECT *
  FROM ibis_testing.&amp;#96;functional_alltypes&amp;#96;
  LIMIT 20
) t0

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Here's the important thing: &lt;strong&gt;the user never interacts with the actual operations&lt;/strong&gt;. As a user, &lt;strong&gt;all you see are well-typed expressions having a particular value type (scalar, array, or table)&lt;/strong&gt;. This makes your mode of interaction consistent and 100% reliable from step to step.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="how-operations-yield-expressions"&gt;How operations yield expressions&lt;/h2&gt;
&lt;p&gt;To help understand how the DSL is constructed, let's deconstruct a simple string operation:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[18]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;literal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;wes mckinney&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[18]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.StringScalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[19]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;chopped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;substr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;chopped&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[19]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
Substring[string]
  Literal[string]
    wes mckinney
  Literal[int8]
    4
  None
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[20]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;chopped&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[20]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
&amp;apos;mckinney&amp;apos;
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Here is the implementation of &lt;code&gt;Substring&lt;/code&gt;:&lt;/p&gt;
&lt;pre class="sourceCode python"&gt;&lt;code class="sourceCode python"&gt;&lt;span class="kw"&gt;class&lt;/span&gt; Substring(ValueOp):

    input_type = [string, integer(name=&lt;span class="st"&gt;&amp;#39;start&amp;#39;&lt;/span&gt;),
                  integer(name=&lt;span class="st"&gt;&amp;#39;length&amp;#39;&lt;/span&gt;, optional=&lt;span class="ot"&gt;True&lt;/span&gt;)]
    output_type = rules.shape_like_arg(&lt;span class="dv"&gt;0&lt;/span&gt;, &lt;span class="st"&gt;&amp;#39;string&amp;#39;&lt;/span&gt;)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;What's going on when I call &lt;code&gt;expr.substr(4)&lt;/code&gt; is, in fact:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[21]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis.expr.operations&lt;/span&gt; &lt;span class="kn"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;ops&lt;/span&gt;

&lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ops&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Substring&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[21]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.operations.Substring
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Among other things, this validated that the arguments match the &lt;code&gt;input_type&lt;/code&gt; signature.&lt;/p&gt;
&lt;p&gt;The user will &lt;em&gt;never&lt;/em&gt; directly interact with this object. All that matters is that its output is another string.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[22]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_type&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[22]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.StringScalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Internally, Ibis calls the operation instance's &lt;code&gt;to_expr&lt;/code&gt; method which wraps the operation in a &lt;code&gt;StringScalar&lt;/code&gt; expression:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[23]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;chopped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to_expr&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;chopped&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[23]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
Substring[string]
  Literal[string]
    wes mckinney
  Literal[int8]
    4
  None
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="helping-you-avoid-invalid-operations"&gt;Helping you avoid invalid operations&lt;/h2&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;NB.&lt;/strong&gt; This addresses a frequent rough edge I experienced when using NumPy to build pandas; namely that there is a single &lt;code&gt;ndarray&lt;/code&gt; type whose values are only distinguished by the &lt;code&gt;dtype&lt;/code&gt; attribute. pandas is now full of workarounds to provide data type-specific functionality on &lt;code&gt;Series&lt;/code&gt; objects.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Ibis tries to save you from forming invalid expressions in two ways:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;All instance methods on an expression are guaranteed to be valid (assuming there is a translation rule for your target SQL engine).&lt;/li&gt;
&lt;li&gt;Type-checking happens immediately, informing you of any mistakes&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Let's have a closer look at this. I'll use a simplified table (not referencing any real data) to make the output here more compact:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[24]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;([(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;v1&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;double&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                 &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;v2&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;int32&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                 &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;v3&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;string&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;hamster&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;t2&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[24]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
UnboundTable[table]
  name: hamster
  schema:
    v1 : double
    v2 : int32
    v3 : string
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's look at the string column &lt;code&gt;v3&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[25]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[25]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ref_0
UnboundTable[table]
  name: hamster
  schema:
    v1 : double
    v2 : int32
    v3 : string

v3 = Column[array(string)] &amp;apos;v3&amp;apos; from table ref_0
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;The numeric methods we used above are not even available on v3:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[26]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyerr"&gt;
&lt;pre&gt;
&lt;span class="ansired"&gt;---------------------------------------------------------------------------&lt;/span&gt;
&lt;span class="ansired"&gt;AttributeError&lt;/span&gt;                            Traceback (most recent call last)
&lt;span class="ansigreen"&gt;&amp;lt;ipython-input-26-fbd74f99b121&amp;gt;&lt;/span&gt; in &lt;span class="ansicyan"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;&lt;span class="ansiblue"&gt;()&lt;/span&gt;
&lt;span class="ansigreen"&gt;----&amp;gt; 1&lt;/span&gt;&lt;span class="ansired"&gt; &lt;/span&gt;t2&lt;span class="ansiblue"&gt;.&lt;/span&gt;v3&lt;span class="ansiblue"&gt;.&lt;/span&gt;sum&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansired"&gt;AttributeError&lt;/span&gt;: &amp;apos;StringArray&amp;apos; object has no attribute &amp;apos;sum&amp;apos;&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This constraint applies to all expressions. Suppose we casted &lt;code&gt;v1&lt;/code&gt; to string; this would still be true:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[27]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[27]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ref_0
UnboundTable[table]
  name: hamster
  schema:
    v1 : double
    v2 : int32
    v3 : string

sum = Sum[double]
  v1 = Column[array(double)] &amp;apos;v1&amp;apos; from table ref_0
  None
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[28]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;string&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyerr"&gt;
&lt;pre&gt;
&lt;span class="ansired"&gt;---------------------------------------------------------------------------&lt;/span&gt;
&lt;span class="ansired"&gt;AttributeError&lt;/span&gt;                            Traceback (most recent call last)
&lt;span class="ansigreen"&gt;&amp;lt;ipython-input-28-b4e33ca864fd&amp;gt;&lt;/span&gt; in &lt;span class="ansicyan"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;&lt;span class="ansiblue"&gt;()&lt;/span&gt;
&lt;span class="ansigreen"&gt;----&amp;gt; 1&lt;/span&gt;&lt;span class="ansired"&gt; &lt;/span&gt;t2&lt;span class="ansiblue"&gt;.&lt;/span&gt;v1&lt;span class="ansiblue"&gt;.&lt;/span&gt;cast&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;&amp;apos;string&amp;apos;&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;.&lt;/span&gt;sum&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansired"&gt;AttributeError&lt;/span&gt;: &amp;apos;StringArray&amp;apos; object has no attribute &amp;apos;sum&amp;apos;&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;In addition to not allowing you to call invalid instance methods, input arguments are all type-checked:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[29]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# valid&lt;/span&gt;

&lt;span class="n"&gt;t2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;foo&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# invalid&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyerr"&gt;
&lt;pre&gt;
&lt;span class="ansired"&gt;---------------------------------------------------------------------------&lt;/span&gt;
&lt;span class="ansired"&gt;IbisTypeError&lt;/span&gt;                             Traceback (most recent call last)
&lt;span class="ansigreen"&gt;&amp;lt;ipython-input-29-844feacfdc97&amp;gt;&lt;/span&gt; in &lt;span class="ansicyan"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;&lt;span class="ansiblue"&gt;()&lt;/span&gt;
&lt;span class="ansigreen"&gt;      1&lt;/span&gt; expr &lt;span class="ansiblue"&gt;=&lt;/span&gt; t2&lt;span class="ansiblue"&gt;.&lt;/span&gt;v1&lt;span class="ansiblue"&gt;.&lt;/span&gt;round&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansicyan"&gt;2&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;  &lt;span class="ansired"&gt;# valid&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;      2&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;----&amp;gt; 3&lt;/span&gt;&lt;span class="ansired"&gt; &lt;/span&gt;t2&lt;span class="ansiblue"&gt;.&lt;/span&gt;v1&lt;span class="ansiblue"&gt;.&lt;/span&gt;round&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;&amp;apos;foo&amp;apos;&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;  &lt;span class="ansired"&gt;# invalid&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/code/ibis/ibis/expr/api.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;round&lt;/span&gt;&lt;span class="ansiblue"&gt;(arg, digits)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    955&lt;/span&gt;         other numeric types&lt;span class="ansiblue"&gt;:&lt;/span&gt; double&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    956&lt;/span&gt;     &amp;quot;&amp;quot;&amp;quot;
&lt;span class="ansigreen"&gt;--&amp;gt; 957&lt;/span&gt;&lt;span class="ansired"&gt;     &lt;/span&gt;op &lt;span class="ansiblue"&gt;=&lt;/span&gt; _ops&lt;span class="ansiblue"&gt;.&lt;/span&gt;Round&lt;span class="ansiblue"&gt;(&lt;/span&gt;arg&lt;span class="ansiblue"&gt;,&lt;/span&gt; digits&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    958&lt;/span&gt;     &lt;span class="ansigreen"&gt;return&lt;/span&gt; op&lt;span class="ansiblue"&gt;.&lt;/span&gt;to_expr&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    959&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/code/ibis/ibis/expr/types.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;__init__&lt;/span&gt;&lt;span class="ansiblue"&gt;(self, *args)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    269&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    270&lt;/span&gt;     &lt;span class="ansigreen"&gt;def&lt;/span&gt; __init__&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;,&lt;/span&gt; &lt;span class="ansiblue"&gt;*&lt;/span&gt;args&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;--&amp;gt; 271&lt;/span&gt;&lt;span class="ansired"&gt;         &lt;/span&gt;args &lt;span class="ansiblue"&gt;=&lt;/span&gt; self&lt;span class="ansiblue"&gt;.&lt;/span&gt;_validate_args&lt;span class="ansiblue"&gt;(&lt;/span&gt;args&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    272&lt;/span&gt;         Node&lt;span class="ansiblue"&gt;.&lt;/span&gt;__init__&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;,&lt;/span&gt; args&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    273&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/code/ibis/ibis/expr/types.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;_validate_args&lt;/span&gt;&lt;span class="ansiblue"&gt;(self, args)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    276&lt;/span&gt;             &lt;span class="ansigreen"&gt;return&lt;/span&gt; args&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    277&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;--&amp;gt; 278&lt;/span&gt;&lt;span class="ansired"&gt;         &lt;/span&gt;&lt;span class="ansigreen"&gt;return&lt;/span&gt; self&lt;span class="ansiblue"&gt;.&lt;/span&gt;input_type&lt;span class="ansiblue"&gt;.&lt;/span&gt;validate&lt;span class="ansiblue"&gt;(&lt;/span&gt;args&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    279&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    280&lt;/span&gt;     &lt;span class="ansigreen"&gt;def&lt;/span&gt; root_tables&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/code/ibis/ibis/expr/rules.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;validate&lt;/span&gt;&lt;span class="ansiblue"&gt;(self, args)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    345&lt;/span&gt;             args &lt;span class="ansiblue"&gt;=&lt;/span&gt; list&lt;span class="ansiblue"&gt;(&lt;/span&gt;args&lt;span class="ansiblue"&gt;)&lt;/span&gt; &lt;span class="ansiblue"&gt;+&lt;/span&gt; &lt;span class="ansiblue"&gt;[&lt;/span&gt;t&lt;span class="ansiblue"&gt;.&lt;/span&gt;default &lt;span class="ansigreen"&gt;for&lt;/span&gt; t &lt;span class="ansigreen"&gt;in&lt;/span&gt; self&lt;span class="ansiblue"&gt;.&lt;/span&gt;types&lt;span class="ansiblue"&gt;[&lt;/span&gt;n&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;]&lt;/span&gt;&lt;span class="ansiblue"&gt;]&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    346&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;--&amp;gt; 347&lt;/span&gt;&lt;span class="ansired"&gt;         &lt;/span&gt;&lt;span class="ansigreen"&gt;return&lt;/span&gt; self&lt;span class="ansiblue"&gt;.&lt;/span&gt;_validate&lt;span class="ansiblue"&gt;(&lt;/span&gt;args&lt;span class="ansiblue"&gt;,&lt;/span&gt; self&lt;span class="ansiblue"&gt;.&lt;/span&gt;types&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    348&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    349&lt;/span&gt;     &lt;span class="ansigreen"&gt;def&lt;/span&gt; _validate&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;,&lt;/span&gt; args&lt;span class="ansiblue"&gt;,&lt;/span&gt; types&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/code/ibis/ibis/expr/rules.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;_validate&lt;/span&gt;&lt;span class="ansiblue"&gt;(self, args, types)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    356&lt;/span&gt;                 msg = (&amp;apos;Argument {0}: {1}&amp;apos;.format(i, exc) +
&lt;span class="ansigreen"&gt;    357&lt;/span&gt;                        &amp;apos;\nArgument was: {0}&amp;apos;.format(ir._safe_repr(args[i])))
&lt;span class="ansigreen"&gt;--&amp;gt; 358&lt;/span&gt;&lt;span class="ansired"&gt;                 &lt;/span&gt;&lt;span class="ansigreen"&gt;raise&lt;/span&gt; IbisTypeError&lt;span class="ansiblue"&gt;(&lt;/span&gt;msg&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    359&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    360&lt;/span&gt;         &lt;span class="ansigreen"&gt;return&lt;/span&gt; clean_args&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansired"&gt;IbisTypeError&lt;/span&gt;: Argument 1: not integer
Argument was: &amp;apos;foo&amp;apos;&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="expression-type-hierarchies"&gt;Expression type hierarchies&lt;/h2&gt;
&lt;p&gt;The user expression types inherit from a particular value type along with a shape type (table, array, or scalar).&lt;/p&gt;
&lt;center&gt;
&lt;img src="../images/int8value-mro.png"&gt;&lt;/img&gt;
&lt;/center&gt;

&lt;p&gt;Methods are shared amongst type classes, and the classes assist with type validation, implicit casts, and other internal matters in the DSL and SQL compiler toolchain.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="some-ibis-and-blaze-dsl-comparisons"&gt;Some Ibis and Blaze DSL comparisons&lt;/h2&gt;
&lt;p&gt;Since DSLs are inherently nuanced things, the only real way to evaluate them fully is to try to solve &lt;em&gt;your problem&lt;/em&gt; using one of them.&lt;/p&gt;
&lt;p&gt;Architectural similarities are interesting, but in my experience &lt;strong&gt;API design / usability&lt;/strong&gt; and &lt;strong&gt;comprehensive solutions to use cases&lt;/strong&gt; are the only things that users care about. Blaze today conflicts with my (definitely opinionated) API design preferences and my understanding of the needs of SQL users (the intended audience for Ibis). It was much simpler for me to design a solution from scratch (whose design can be quickly changed without project governance issues) that meets my target users' design and usability requirements.&lt;/p&gt;
&lt;p&gt;In an &lt;a href="http://blog.ibis-project.org/other-compute-engines/"&gt;earlier post here&lt;/a&gt;, I described how our priorities with Ibis are out of alignment with Blaze. I had a phone call with Continuum Analytics this past week to discuss Ibis and Blaze goals, use cases, and possible collaborations. They asked me for specific feedback on Blaze's API, so I'm digging in here to look closer.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[30]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;blaze&lt;/span&gt; &lt;span class="kn"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;bz&lt;/span&gt;

&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bz&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TableSymbol&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;hamster&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;{v1: float64, v2: int32, v3: string}&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;t&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[30]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
hamster
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[31]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[31]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
blaze.expr.expressions.Symbol
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Okay, how about the class hierarchy?&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[32]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mro&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[32]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[blaze.expr.expressions.Symbol,
 blaze.expr.expressions.Expr,
 blaze.expr.core.Node,
 object]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Alright. You can find out what kind of expression you're dealing with by looking at the &lt;code&gt;dshape&lt;/code&gt; attribute:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[33]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dshape&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[33]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
dshape(&amp;quot;var * {v1: float64, v2: int32, v3: string}&amp;quot;)
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This is very NumPy-like, which is good and bad for reasons I've cited.&lt;/p&gt;
&lt;p&gt;Let's look at a few simple expressions like the above:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[34]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;In Ibis, this yields a &lt;code&gt;DoubleScalar&lt;/code&gt;. Here it's a little different because there are no value types, only &lt;code&gt;Expr&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[35]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[35]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
sum(hamster.v1)
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[36]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[36]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
blaze.expr.reductions.sum
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[37]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mro&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[37]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[blaze.expr.reductions.sum,
 blaze.expr.reductions.Reduction,
 blaze.expr.expressions.Expr,
 blaze.expr.core.Node,
 object]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[38]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dshape&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[38]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
dshape(&amp;quot;float64&amp;quot;)
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;You also can't do &lt;code&gt;sum&lt;/code&gt; on a string:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[39]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyerr"&gt;
&lt;pre&gt;
&lt;span class="ansired"&gt;---------------------------------------------------------------------------&lt;/span&gt;
&lt;span class="ansired"&gt;AttributeError&lt;/span&gt;                            Traceback (most recent call last)
&lt;span class="ansigreen"&gt;&amp;lt;ipython-input-39-32d7bb6499a7&amp;gt;&lt;/span&gt; in &lt;span class="ansicyan"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;&lt;span class="ansiblue"&gt;()&lt;/span&gt;
&lt;span class="ansigreen"&gt;----&amp;gt; 1&lt;/span&gt;&lt;span class="ansired"&gt; &lt;/span&gt;t&lt;span class="ansiblue"&gt;.&lt;/span&gt;v3&lt;span class="ansiblue"&gt;.&lt;/span&gt;sum&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/anaconda/envs/ibis-blog/lib/python2.7/site-packages/blaze/expr/expressions.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;__getattr__&lt;/span&gt;&lt;span class="ansiblue"&gt;(self, key)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    168&lt;/span&gt;             &lt;span class="ansigreen"&gt;pass&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    169&lt;/span&gt;         &lt;span class="ansigreen"&gt;try&lt;/span&gt;&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;--&amp;gt; 170&lt;/span&gt;&lt;span class="ansired"&gt;             &lt;/span&gt;result &lt;span class="ansiblue"&gt;=&lt;/span&gt; object&lt;span class="ansiblue"&gt;.&lt;/span&gt;__getattribute__&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;,&lt;/span&gt; key&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    171&lt;/span&gt;         &lt;span class="ansigreen"&gt;except&lt;/span&gt; AttributeError&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    172&lt;/span&gt;             fields = dict(zip(map(valid_identifier, self.fields),

&lt;span class="ansired"&gt;AttributeError&lt;/span&gt;: &amp;apos;Field&amp;apos; object has no attribute &amp;apos;sum&amp;apos;&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Most methods in Blaze are not instance methods, and are instead found in &lt;code&gt;blaze.expr.functions&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[40]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;blaze.expr.functions&lt;/span&gt; &lt;span class="kn"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;bzfunc&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Sadly, things start going poorly when I make mistakes using functions in &lt;code&gt;bzfunc&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[41]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bzfunc&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[41]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
sum(hamster.v3)
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Wait, wasn't &lt;code&gt;v3&lt;/code&gt; a string?&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[42]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dshape&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyerr"&gt;
&lt;pre&gt;
&lt;span class="ansired"&gt;---------------------------------------------------------------------------&lt;/span&gt;
&lt;span class="ansired"&gt;AssertionError&lt;/span&gt;                            Traceback (most recent call last)
&lt;span class="ansigreen"&gt;&amp;lt;ipython-input-42-b75fb756f813&amp;gt;&lt;/span&gt; in &lt;span class="ansicyan"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;&lt;span class="ansiblue"&gt;()&lt;/span&gt;
&lt;span class="ansigreen"&gt;----&amp;gt; 1&lt;/span&gt;&lt;span class="ansired"&gt; &lt;/span&gt;expr&lt;span class="ansiblue"&gt;.&lt;/span&gt;dshape&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/anaconda/envs/ibis-blog/lib/python2.7/site-packages/blaze/expr/reductions.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;dshape&lt;/span&gt;&lt;span class="ansiblue"&gt;(self)&lt;/span&gt;
&lt;span class="ansigreen"&gt;     64&lt;/span&gt;                           &lt;span class="ansigreen"&gt;for&lt;/span&gt; i&lt;span class="ansiblue"&gt;,&lt;/span&gt; d &lt;span class="ansigreen"&gt;in&lt;/span&gt; enumerate&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;.&lt;/span&gt;_child&lt;span class="ansiblue"&gt;.&lt;/span&gt;shape&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;     65&lt;/span&gt;                           if i not in axis)
&lt;span class="ansigreen"&gt;---&amp;gt; 66&lt;/span&gt;&lt;span class="ansired"&gt;         &lt;/span&gt;&lt;span class="ansigreen"&gt;return&lt;/span&gt; DataShape&lt;span class="ansiblue"&gt;(&lt;/span&gt;&lt;span class="ansiblue"&gt;*&lt;/span&gt;&lt;span class="ansiblue"&gt;(&lt;/span&gt;shape &lt;span class="ansiblue"&gt;+&lt;/span&gt; &lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;.&lt;/span&gt;schema&lt;span class="ansiblue"&gt;,&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;     67&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;     68&lt;/span&gt;     &lt;span class="ansiblue"&gt;@&lt;/span&gt;property&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/anaconda/envs/ibis-blog/lib/python2.7/site-packages/blaze/expr/reductions.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;schema&lt;/span&gt;&lt;span class="ansiblue"&gt;(self)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    115&lt;/span&gt;     &lt;span class="ansiblue"&gt;@&lt;/span&gt;property&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    116&lt;/span&gt;     &lt;span class="ansigreen"&gt;def&lt;/span&gt; schema&lt;span class="ansiblue"&gt;(&lt;/span&gt;self&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;--&amp;gt; 117&lt;/span&gt;&lt;span class="ansired"&gt;         &lt;/span&gt;&lt;span class="ansigreen"&gt;return&lt;/span&gt; DataShape&lt;span class="ansiblue"&gt;(&lt;/span&gt;datashape&lt;span class="ansiblue"&gt;.&lt;/span&gt;maxtype&lt;span class="ansiblue"&gt;(&lt;/span&gt;super&lt;span class="ansiblue"&gt;(&lt;/span&gt;sum&lt;span class="ansiblue"&gt;,&lt;/span&gt; self&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;.&lt;/span&gt;schema&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    118&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    119&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansigreen"&gt;/Users/wes/anaconda/envs/ibis-blog/lib/python2.7/site-packages/datashape/typesets.pyc&lt;/span&gt; in &lt;span class="ansicyan"&gt;maxtype&lt;/span&gt;&lt;span class="ansiblue"&gt;(measure)&lt;/span&gt;
&lt;span class="ansigreen"&gt;    182&lt;/span&gt;     &lt;span class="ansigreen"&gt;if&lt;/span&gt; isoption&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    183&lt;/span&gt;         measure &lt;span class="ansiblue"&gt;=&lt;/span&gt; measure&lt;span class="ansiblue"&gt;.&lt;/span&gt;ty&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;--&amp;gt; 184&lt;/span&gt;&lt;span class="ansired"&gt;     &lt;/span&gt;&lt;span class="ansigreen"&gt;assert&lt;/span&gt; matches_typeset&lt;span class="ansiblue"&gt;(&lt;/span&gt;measure&lt;span class="ansiblue"&gt;,&lt;/span&gt; scalar&lt;span class="ansiblue"&gt;)&lt;/span&gt;&lt;span class="ansiblue"&gt;,&lt;/span&gt; &lt;span class="ansiblue"&gt;&amp;apos;measure must be numeric&amp;apos;&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    185&lt;/span&gt; &lt;span class="ansiblue"&gt;&lt;/span&gt;
&lt;span class="ansigreen"&gt;    186&lt;/span&gt;     &lt;span class="ansigreen"&gt;if&lt;/span&gt; measure &lt;span class="ansiblue"&gt;==&lt;/span&gt; bool_&lt;span class="ansiblue"&gt;:&lt;/span&gt;&lt;span class="ansiblue"&gt;&lt;/span&gt;

&lt;span class="ansired"&gt;AssertionError&lt;/span&gt;: measure must be numeric&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let me try something else, like &lt;code&gt;sqrt&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[43]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bzfunc&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dshape&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[43]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
dshape(&amp;quot;var * float64&amp;quot;)
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;So that's not good either. I couldn't find a &lt;code&gt;round&lt;/code&gt; function. Let me try adding a double to a string:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[44]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[44]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
hamster.v1 + hamster.v3
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;I don't think that should work either. Also:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[45]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dshape&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[45]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
dshape(&amp;quot;var * object&amp;quot;)
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Invalid table-level expressions are also not problematic at all:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[46]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v3&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;I also find the spread of functionality between &lt;code&gt;blaze.expr.functions&lt;/code&gt; and instance methods to conflict with my sensibilities (which are admittedly highly subjective and pandas-influenced). For example, a float scalar object has some functions but others are found elsewhere.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[47]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bzfunc&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dshape&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;dir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;_&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
float64

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[47]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[&amp;apos;apply&amp;apos;,
 &amp;apos;axis&amp;apos;,
 &amp;apos;coerce&amp;apos;,
 &amp;apos;dshape&amp;apos;,
 &amp;apos;fields&amp;apos;,
 &amp;apos;isidentical&amp;apos;,
 &amp;apos;isnan&amp;apos;,
 &amp;apos;keepdims&amp;apos;,
 &amp;apos;label&amp;apos;,
 &amp;apos;map&amp;apos;,
 &amp;apos;relabel&amp;apos;,
 &amp;apos;schema&amp;apos;,
 &amp;apos;symbol&amp;apos;,
 &amp;apos;truncate&amp;apos;,
 &amp;apos;utcfromtimestamp&amp;apos;]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;It's really just a difference in taste, but I prefer to have all my valid functions available as instance methods so that I can compose function chains more easily as seen above.&lt;/p&gt;
&lt;p&gt;I wanted to try to reproduce examples on top of SQLite from my &lt;a href="http://blog.ibis-project.org/sqlite-crunchbase-quickstart/"&gt;last blog post&lt;/a&gt; but I found too much missing functionality, to name a couple specific things:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;case statements / if-else expressions. This also makes &lt;code&gt;bucket&lt;/code&gt; not possible to implement&lt;/li&gt;
&lt;li&gt;&lt;code&gt;fillna&lt;/code&gt; function&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Also, the way that joins work in Blaze is very un-SQL like (replicates pandas, in essence). This makes many SQL queries with joins hard to port. Check out the &lt;a href="http://docs.ibis-project.org/tutorial.html"&gt;Ibis tutorial&lt;/a&gt; for more on that.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Quickstart on Crunchbase analysis using Ibis and SQLite</title><link href="http://www.ibis-project.org/sqlite-crunchbase-quickstart/" rel="alternate"></link><updated>2015-09-09T08:01:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-09-09:sqlite-crunchbase-quickstart/</id><summary type="html">

&lt;p&gt;The latest &lt;a href="http://blog.ibis-project.org/release-0.5/" target="_blank"&gt;0.5 version&lt;/a&gt; of Ibis supports executing
expressions on data in SQLite database files. Since SQLite ships with Python,
this is the easiest way to get up and running using the project because you
don't have to install or administer a database at all.&lt;/p&gt;
&lt;p&gt;To get going, install Ibis if you have not already&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;pip install ibis-framework
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;And download the &lt;a href="https://ibis-resources.s3.amazonaws.com/data/crunchbase/crunchbase.db"&gt;Crunchbase dataset&lt;/a&gt; found on the &lt;a href="http://blog.ibis-project.org/pages/data.html" target="_blank"&gt;data collection page&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Now fire up a IPython / Jupyter notebook or shell and let's get started!&lt;/p&gt;


&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To get started, import &lt;code&gt;ibis&lt;/code&gt;, set it to interactive mode, and open the SQLite file you just downloaded like so:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;
&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactive&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;

&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sqlite&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;crunchbase.db&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;The &lt;code&gt;list_tables&lt;/code&gt; method shows you what tables are in the file:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list_tables&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[2]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
[u&amp;apos;acquisitions&amp;apos;, u&amp;apos;companies&amp;apos;, u&amp;apos;investments&amp;apos;, u&amp;apos;rounds&amp;apos;]
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's look at the &lt;code&gt;rounds&lt;/code&gt; table:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;rounds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;rounds&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
Table rows: 87161

Column                   Type    Non-null #
------                   ----    ----------
company_permalink        string  87161     
company_name             string  87161     
company_category_list    string  83464     
company_market           string  80523     
company_country_code     string  81291     
company_state_code       string  59296     
company_region           string  81016     
company_city             string  80208     
funding_round_permalink  string  87161     
funding_round_type       string  87161     
funding_round_code       string  23648     
funded_at                string  87161     
funded_month             string  87147     
funded_quarter           string  87147     
funded_year              int64   87147     
raised_amount_usd        double  73406     
&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;We can start to explore this data set using pandas-like syntax:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_round_type&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
       funding_round_type  count
0                   angel   4602
1        convertible_note    838
2          debt_financing   5869
3     equity_crowdfunding   2401
4                   grant   1523
5   non_equity_assistance     69
6           post_ipo_debt     92
7         post_ipo_equity    425
8          private_equity   1864
9    product_crowdfunding    241
10       secondary_market     61
11                   seed  22053
12            undisclosed   4128
13                venture  42995
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;There's too little space here to give a comprehensive tutorial on the Ibis DSL, but it's designed for composability and operation chaining even more so than pandas.&lt;/p&gt;
&lt;p&gt;For example, we could find out how many acquisitions had a known price as follows&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;acquisitions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;acquisitions&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;acquisitions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;price_amount&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isnull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;has_price&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[5]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   has_price  count
0          0   3816
1          1  51424
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;If you call the &lt;code&gt;execute&lt;/code&gt; method on the expression, you'll get a pandas object back:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[6]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;df&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[6]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;has_price&lt;/th&gt;
      &lt;th&gt;count&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;3816&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;51424&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Ibis has three basic data types:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Tables&lt;/li&gt;
&lt;li&gt;Arrays&lt;/li&gt;
&lt;li&gt;Scalars&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Based on the type of the expression, you'll get back a pandas or Python object that is an in-memory counterpart&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[7]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[7]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.TableExpr
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[8]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;companies&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_total_usd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[8]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.types.DoubleScalar
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[9]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[9]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
16198405.416388474
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="a-few-funding-metrics"&gt;A few funding metrics&lt;/h2&gt;
&lt;p&gt;Let's suppose we were interested in looking at the number of coded (Series A, Series B, etc.) venture investments in the dataset for each year.&lt;/p&gt;
&lt;p&gt;First, a technical detail. SQLite stores all date and time values as strings, but Ibis lets you cast strings to timestamps, handling the rest of the details for you&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[10]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;funded_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funded_at&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;timestamp&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;funded_at&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[10]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
    unnamed  count
0         1      3
1         7      1
2        11      1
3        20      2
4        29      1
5        30      1
6       201      1
7       203      1
8       214      1
9      1921      1
10     1960      3
11     1973      1
12     1974      2
13     1979      1
14     1982      3
15     1983      1
16     1984      4
17     1985      6
18     1986      4
19     1987      6
20     1988      2
21     1989      4
22     1990     14
23     1991     10
24     1992     12
25     1993     15
26     1994     15
27     1995     21
28     1996     32
29     1997     30
30     1998     62
31     1999    178
32     2000    267
33     2001    200
34     2002    207
35     2003    271
36     2004    434
37     2005   1762
38     2006   2663
39     2007   3562
40     2008   4257
41     2009   6553
42     2010   8692
43     2011  10550
44     2012  12648
45     2013  16194
46     2014  17670
47     2015    790
48     2104      1
49     2105      1
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This looks like some bad data, so I'll restrict the analysis to 2000 to 2015. Also, there are NULL values in &lt;code&gt;funding_round_code&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[11]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_round_code&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[11]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
  funding_round_code  count
0               None  63513
1                  A  11382
2                  B   6548
3                  C   3329
4                  D   1530
5                  E    608
6                  F    201
7                  G     45
8                  H      5
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[12]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;funded_at&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_round_type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;venture&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
               &lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;between&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2015&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
               &lt;span class="n"&gt;rounds&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_round_code&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notnull&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;funding_round_code&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[12]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;year&lt;/th&gt;
      &lt;th&gt;funding_round_code&lt;/th&gt;
      &lt;th&gt;count&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;A&lt;/td&gt;
      &lt;td&gt;79&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;B&lt;/td&gt;
      &lt;td&gt;32&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;C&lt;/td&gt;
      &lt;td&gt;10&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;D&lt;/td&gt;
      &lt;td&gt;8&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;E&lt;/td&gt;
      &lt;td&gt;2&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;5&lt;/th&gt;
      &lt;td&gt;2001&lt;/td&gt;
      &lt;td&gt;A&lt;/td&gt;
      &lt;td&gt;50&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;6&lt;/th&gt;
      &lt;td&gt;2001&lt;/td&gt;
      &lt;td&gt;B&lt;/td&gt;
      &lt;td&gt;30&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;7&lt;/th&gt;
      &lt;td&gt;2001&lt;/td&gt;
      &lt;td&gt;C&lt;/td&gt;
      &lt;td&gt;17&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;8&lt;/th&gt;
      &lt;td&gt;2001&lt;/td&gt;
      &lt;td&gt;D&lt;/td&gt;
      &lt;td&gt;4&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;9&lt;/th&gt;
      &lt;td&gt;2001&lt;/td&gt;
      &lt;td&gt;E&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Since &lt;code&gt;results&lt;/code&gt; is now a pandas DataFrame, it can be reshaped accordingly:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[13]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;pivoted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_index&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;funding_round_code&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
           &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unstack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;funding_round_code&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
           &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;pivoted&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[13]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th colspan="8" halign="left"&gt;count&lt;/th&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;funding_round_code&lt;/th&gt;
      &lt;th&gt;A&lt;/th&gt;
      &lt;th&gt;B&lt;/th&gt;
      &lt;th&gt;C&lt;/th&gt;
      &lt;th&gt;D&lt;/th&gt;
      &lt;th&gt;E&lt;/th&gt;
      &lt;th&gt;F&lt;/th&gt;
      &lt;th&gt;G&lt;/th&gt;
      &lt;th&gt;H&lt;/th&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;year&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;2000&lt;/th&gt;
      &lt;td&gt;79&lt;/td&gt;
      &lt;td&gt;32&lt;/td&gt;
      &lt;td&gt;10&lt;/td&gt;
      &lt;td&gt;8&lt;/td&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2001&lt;/th&gt;
      &lt;td&gt;50&lt;/td&gt;
      &lt;td&gt;30&lt;/td&gt;
      &lt;td&gt;17&lt;/td&gt;
      &lt;td&gt;4&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2002&lt;/th&gt;
      &lt;td&gt;35&lt;/td&gt;
      &lt;td&gt;39&lt;/td&gt;
      &lt;td&gt;25&lt;/td&gt;
      &lt;td&gt;5&lt;/td&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2003&lt;/th&gt;
      &lt;td&gt;68&lt;/td&gt;
      &lt;td&gt;45&lt;/td&gt;
      &lt;td&gt;14&lt;/td&gt;
      &lt;td&gt;13&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2004&lt;/th&gt;
      &lt;td&gt;146&lt;/td&gt;
      &lt;td&gt;76&lt;/td&gt;
      &lt;td&gt;32&lt;/td&gt;
      &lt;td&gt;15&lt;/td&gt;
      &lt;td&gt;3&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2005&lt;/th&gt;
      &lt;td&gt;513&lt;/td&gt;
      &lt;td&gt;334&lt;/td&gt;
      &lt;td&gt;176&lt;/td&gt;
      &lt;td&gt;67&lt;/td&gt;
      &lt;td&gt;26&lt;/td&gt;
      &lt;td&gt;6&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2006&lt;/th&gt;
      &lt;td&gt;717&lt;/td&gt;
      &lt;td&gt;465&lt;/td&gt;
      &lt;td&gt;226&lt;/td&gt;
      &lt;td&gt;91&lt;/td&gt;
      &lt;td&gt;35&lt;/td&gt;
      &lt;td&gt;7&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2007&lt;/th&gt;
      &lt;td&gt;956&lt;/td&gt;
      &lt;td&gt;583&lt;/td&gt;
      &lt;td&gt;281&lt;/td&gt;
      &lt;td&gt;110&lt;/td&gt;
      &lt;td&gt;49&lt;/td&gt;
      &lt;td&gt;7&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2008&lt;/th&gt;
      &lt;td&gt;979&lt;/td&gt;
      &lt;td&gt;653&lt;/td&gt;
      &lt;td&gt;308&lt;/td&gt;
      &lt;td&gt;120&lt;/td&gt;
      &lt;td&gt;54&lt;/td&gt;
      &lt;td&gt;17&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2009&lt;/th&gt;
      &lt;td&gt;753&lt;/td&gt;
      &lt;td&gt;531&lt;/td&gt;
      &lt;td&gt;290&lt;/td&gt;
      &lt;td&gt;147&lt;/td&gt;
      &lt;td&gt;55&lt;/td&gt;
      &lt;td&gt;28&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2010&lt;/th&gt;
      &lt;td&gt;1013&lt;/td&gt;
      &lt;td&gt;598&lt;/td&gt;
      &lt;td&gt;369&lt;/td&gt;
      &lt;td&gt;149&lt;/td&gt;
      &lt;td&gt;52&lt;/td&gt;
      &lt;td&gt;18&lt;/td&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2011&lt;/th&gt;
      &lt;td&gt;1250&lt;/td&gt;
      &lt;td&gt;700&lt;/td&gt;
      &lt;td&gt;334&lt;/td&gt;
      &lt;td&gt;175&lt;/td&gt;
      &lt;td&gt;60&lt;/td&gt;
      &lt;td&gt;18&lt;/td&gt;
      &lt;td&gt;5&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2012&lt;/th&gt;
      &lt;td&gt;1242&lt;/td&gt;
      &lt;td&gt;610&lt;/td&gt;
      &lt;td&gt;345&lt;/td&gt;
      &lt;td&gt;184&lt;/td&gt;
      &lt;td&gt;69&lt;/td&gt;
      &lt;td&gt;16&lt;/td&gt;
      &lt;td&gt;7&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2013&lt;/th&gt;
      &lt;td&gt;1606&lt;/td&gt;
      &lt;td&gt;796&lt;/td&gt;
      &lt;td&gt;377&lt;/td&gt;
      &lt;td&gt;185&lt;/td&gt;
      &lt;td&gt;81&lt;/td&gt;
      &lt;td&gt;38&lt;/td&gt;
      &lt;td&gt;6&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2014&lt;/th&gt;
      &lt;td&gt;1757&lt;/td&gt;
      &lt;td&gt;952&lt;/td&gt;
      &lt;td&gt;471&lt;/td&gt;
      &lt;td&gt;223&lt;/td&gt;
      &lt;td&gt;108&lt;/td&gt;
      &lt;td&gt;36&lt;/td&gt;
      &lt;td&gt;18&lt;/td&gt;
      &lt;td&gt;5&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2015&lt;/th&gt;
      &lt;td&gt;88&lt;/td&gt;
      &lt;td&gt;71&lt;/td&gt;
      &lt;td&gt;34&lt;/td&gt;
      &lt;td&gt;28&lt;/td&gt;
      &lt;td&gt;8&lt;/td&gt;
      &lt;td&gt;5&lt;/td&gt;
      &lt;td&gt;3&lt;/td&gt;
      &lt;td&gt;0&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Cool. Now let's look at total funding according to some arbitrary buckets. I'll use the &lt;code&gt;bucket&lt;/code&gt; method which computes a discrete histogram according to bin edges you provide:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[14]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;funding_buckets&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;50000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                   &lt;span class="mi"&gt;500000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1000000000&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;
          &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_total_usd&lt;/span&gt;
          &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;funding_buckets&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;include_over&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[14]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   unnamed  count
0      NaN  12055
1        0  15965
2        1  15754
3        2   7839
4        3   1532
5        4   1022
6        5     88
7        6     37
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;These are bucket numbers. I can provide names using the &lt;code&gt;label&lt;/code&gt; method:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[15]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;bucket_names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;0 to 1m&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;1m to 10m&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;10m to 50m&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                &lt;span class="s1"&gt;&amp;#39;50m to 100m&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;100m to 500m&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s1"&gt;&amp;#39;500m to 1b&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Over 1b&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bucket&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;labeled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket_names&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;with_names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;labeled&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;with_names&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[15]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   bucket  count   bucket_name
0     NaN  12055          None
1       0  15965       0 to 1m
2       1  15754     1m to 10m
3       2   7839    10m to 50m
4       3   1532   50m to 100m
5       4   1022  100m to 500m
6       5     88    500m to 1b
7       6     37       Over 1b
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;If you were interested in statistics: other than the count, you can use &lt;code&gt;group_by&lt;/code&gt; and &lt;code&gt;aggregate&lt;/code&gt; to compute multiple group summaries:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[16]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bucket&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
           &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
                      &lt;span class="n"&gt;total_funding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding_total_usd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
           &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket_names&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="n"&gt;metrics&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[16]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   bucket  count  total_funding   bucket_name
0     NaN  12055            NaN          None
1       0  15965   4.505177e+09       0 to 1m
2       1  15754   5.712283e+10     1m to 10m
3       2   7839   1.724166e+11    10m to 50m
4       3   1532   1.054132e+11   50m to 100m
5       4   1022   1.826600e+11  100m to 500m
6       5     88   5.804196e+10    500m to 1b
7       6     37   1.040123e+11       Over 1b
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;I'll embellish this by looking at the operation status of companies founded 2010 onward&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[17]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;joined&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                           &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Unknown&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
          &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;founded_at&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;2010-01-01&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
           &lt;span class="n"&gt;companies&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;founded_at&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isnull&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
          &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bucket&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;status&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
          &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
          &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket_names&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                                         &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Unknown&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))))&lt;/span&gt;

&lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;joined&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_index&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;status&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;bucket&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;bucket_name&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;count&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unstack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;status&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[17]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;status&lt;/th&gt;
      &lt;th&gt;Unknown&lt;/th&gt;
      &lt;th&gt;acquired&lt;/th&gt;
      &lt;th&gt;closed&lt;/th&gt;
      &lt;th&gt;operating&lt;/th&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;bucket&lt;/th&gt;
      &lt;th&gt;bucket_name&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;NaN&lt;/th&gt;
      &lt;th&gt;Unknown&lt;/th&gt;
      &lt;td&gt;3208&lt;/td&gt;
      &lt;td&gt;217&lt;/td&gt;
      &lt;td&gt;251&lt;/td&gt;
      &lt;td&gt;5731&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;th&gt;0 to 1m&lt;/th&gt;
      &lt;td&gt;353&lt;/td&gt;
      &lt;td&gt;227&lt;/td&gt;
      &lt;td&gt;632&lt;/td&gt;
      &lt;td&gt;11195&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;th&gt;1m to 10m&lt;/th&gt;
      &lt;td&gt;197&lt;/td&gt;
      &lt;td&gt;530&lt;/td&gt;
      &lt;td&gt;369&lt;/td&gt;
      &lt;td&gt;8166&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;th&gt;10m to 50m&lt;/th&gt;
      &lt;td&gt;62&lt;/td&gt;
      &lt;td&gt;276&lt;/td&gt;
      &lt;td&gt;124&lt;/td&gt;
      &lt;td&gt;2572&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;th&gt;50m to 100m&lt;/th&gt;
      &lt;td&gt;6&lt;/td&gt;
      &lt;td&gt;47&lt;/td&gt;
      &lt;td&gt;20&lt;/td&gt;
      &lt;td&gt;360&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;th&gt;100m to 500m&lt;/th&gt;
      &lt;td&gt;16&lt;/td&gt;
      &lt;td&gt;14&lt;/td&gt;
      &lt;td&gt;11&lt;/td&gt;
      &lt;td&gt;255&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;5&lt;/th&gt;
      &lt;th&gt;500m to 1b&lt;/th&gt;
      &lt;td&gt;NaN&lt;/td&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;23&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;6&lt;/th&gt;
      &lt;th&gt;Over 1b&lt;/th&gt;
      &lt;td&gt;NaN&lt;/td&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;NaN&lt;/td&gt;
      &lt;td&gt;9&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To explain what all these methods are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;mutate&lt;/code&gt; adds columns to a table&lt;/li&gt;
&lt;li&gt;&lt;code&gt;group_by&lt;/code&gt; creates a grouped table expression that can be aggregated, windowed, or a few other actions&lt;/li&gt;
&lt;li&gt;&lt;code&gt;.size()&lt;/code&gt; is a shortcut for &lt;code&gt;.aggregate(companies.count())&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;fillna&lt;/code&gt; replaces NULL values with the indicated fill value&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To learn more, check out the &lt;a href="http://docs.ibis-project.org/tutorial.html"&gt;tutorial&lt;/a&gt; or the &lt;a href="http://docs.ibis-project.org/api.html"&gt;API reference&lt;/a&gt;. Much more documentation will get written over the coming months, but this gives you an idea of the kind of analysis that's made very easy with Ibis once you learn the DSL.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;By the way, the SQL code that was effectively generated for SQLite by the last expression looks something like this:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[18]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;joined&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt"&gt;&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_stream output_stdout"&gt;
&lt;pre&gt;
SELECT *,
  isnull(CASE &amp;#96;bucket&amp;#96;
    WHEN 0 THEN &amp;apos;0 to 1m&amp;apos;
    WHEN 1 THEN &amp;apos;1m to 10m&amp;apos;
    WHEN 2 THEN &amp;apos;10m to 50m&amp;apos;
    WHEN 3 THEN &amp;apos;50m to 100m&amp;apos;
    WHEN 4 THEN &amp;apos;100m to 500m&amp;apos;
    WHEN 5 THEN &amp;apos;500m to 1b&amp;apos;
    WHEN 6 THEN &amp;apos;Over 1b&amp;apos;
    ELSE NULL
  END, &amp;apos;Unknown&amp;apos;) AS &amp;#96;bucket_name&amp;#96;
FROM (
  SELECT &amp;#96;bucket&amp;#96;, &amp;#96;status&amp;#96;, count(*) AS &amp;#96;count&amp;#96;
  FROM (
    SELECT &amp;#96;permalink&amp;#96;, &amp;#96;name&amp;#96;, &amp;#96;homepage_url&amp;#96;, &amp;#96;category_list&amp;#96;, &amp;#96;market&amp;#96;,
           &amp;#96;funding_total_usd&amp;#96;, isnull(&amp;#96;status&amp;#96;, &amp;apos;Unknown&amp;apos;) AS &amp;#96;status&amp;#96;,
           &amp;#96;country_code&amp;#96;, &amp;#96;state_code&amp;#96;, &amp;#96;region&amp;#96;, &amp;#96;city&amp;#96;, &amp;#96;funding_rounds&amp;#96;,
           &amp;#96;founded_at&amp;#96;, &amp;#96;founded_month&amp;#96;, &amp;#96;founded_quarter&amp;#96;, &amp;#96;founded_year&amp;#96;,
           &amp;#96;first_funding_at&amp;#96;, &amp;#96;last_funding_at&amp;#96;,
      CASE
        WHEN (&amp;#96;funding_total_usd&amp;#96; &amp;gt;= 0) AND (&amp;#96;funding_total_usd&amp;#96; &amp;lt; 1000000) THEN 0
        WHEN (&amp;#96;funding_total_usd&amp;#96; &amp;gt;= 1000000) AND (&amp;#96;funding_total_usd&amp;#96; &amp;lt; 10000000) THEN 1
        WHEN (&amp;#96;funding_total_usd&amp;#96; &amp;gt;= 10000000) AND (&amp;#96;funding_total_usd&amp;#96; &amp;lt; 50000000) THEN 2
        WHEN (&amp;#96;funding_total_usd&amp;#96; &amp;gt;= 50000000) AND (&amp;#96;funding_total_usd&amp;#96; &amp;lt; 100000000) THEN 3
        WHEN (&amp;#96;funding_total_usd&amp;#96; &amp;gt;= 100000000) AND (&amp;#96;funding_total_usd&amp;#96; &amp;lt; 500000000) THEN 4
        WHEN (&amp;#96;funding_total_usd&amp;#96; &amp;gt;= 500000000) AND (&amp;#96;funding_total_usd&amp;#96; &amp;lt;= 1000000000) THEN 5
        WHEN &amp;#96;funding_total_usd&amp;#96; &amp;gt; 1000000000 THEN 6
        ELSE NULL
      END AS &amp;#96;bucket&amp;#96;
    FROM companies
    WHERE (&amp;#96;founded_at&amp;#96; &amp;gt; &amp;apos;2010-01-01&amp;apos;) OR &amp;#96;founded_at&amp;#96; IS NULL
  ) t1
  GROUP BY 1, 2
) t0

&lt;/pre&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Bet you are glad you didn't have to write it by hand!&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="acquisition-rate-by-investor"&gt;Acquisition rate by investor&lt;/h2&gt;
&lt;p&gt;Suppose you were interested in what percentage of a particular venture capital firm's companies eventually get acquired. My friends at Mode Analytics &lt;a href="https://modeanalytics.com/benn/reports/6cd65aab4e18"&gt;solved this problem in SQL&lt;/a&gt; and I reproduce that query here:&lt;/p&gt;
&lt;pre class="sourceCode sql"&gt;&lt;code class="sourceCode sql"&gt;&lt;span class="kw"&gt;SELECT&lt;/span&gt; *,
       acquisitions_ipos/number_of_investments:&lt;span class="ch"&gt;:FLOAT&lt;/span&gt; &lt;span class="kw"&gt;AS&lt;/span&gt; acquitision_rate
  &lt;span class="kw"&gt;FROM&lt;/span&gt; (
&lt;span class="kw"&gt;SELECT&lt;/span&gt; &lt;span class="kw"&gt;CASE&lt;/span&gt; &lt;span class="kw"&gt;WHEN&lt;/span&gt; i.investor_name &lt;span class="kw"&gt;IS&lt;/span&gt; &lt;span class="kw"&gt;NULL&lt;/span&gt; &lt;span class="kw"&gt;THEN&lt;/span&gt; &lt;span class="st"&gt;&amp;#39;NO INVESTOR&amp;#39;&lt;/span&gt;
            &lt;span class="kw"&gt;ELSE&lt;/span&gt; i.investor_name &lt;span class="kw"&gt;END&lt;/span&gt; &lt;span class="ot"&gt;&amp;quot;Investor name&amp;quot;&lt;/span&gt;,
        &lt;span class="fu"&gt;COUNT&lt;/span&gt;(&lt;span class="kw"&gt;DISTINCT&lt;/span&gt; c.permalink) &lt;span class="kw"&gt;AS&lt;/span&gt; number_of_investments,
        &lt;span class="fu"&gt;COUNT&lt;/span&gt;(&lt;span class="kw"&gt;DISTINCT&lt;/span&gt;
        &lt;span class="kw"&gt;CASE&lt;/span&gt; &lt;span class="kw"&gt;WHEN&lt;/span&gt; c.status &lt;span class="kw"&gt;IN&lt;/span&gt; (&lt;span class="st"&gt;&amp;#39;ipo&amp;#39;&lt;/span&gt;, &lt;span class="st"&gt;&amp;#39;acquired&amp;#39;&lt;/span&gt;) &lt;span class="kw"&gt;THEN&lt;/span&gt; c.permalink &lt;span class="kw"&gt;END&lt;/span&gt;) 
        &lt;span class="kw"&gt;AS&lt;/span&gt; acquisitions_ipos
       &lt;span class="kw"&gt;FROM&lt;/span&gt; tutorial.crunchbase_companies c
       &lt;span class="kw"&gt;LEFT&lt;/span&gt; &lt;span class="kw"&gt;JOIN&lt;/span&gt; tutorial.crunchbase_investments i
       &lt;span class="kw"&gt;ON&lt;/span&gt; c.permalink = i.company_permalink
       &lt;span class="kw"&gt;GROUP&lt;/span&gt; &lt;span class="kw"&gt;BY&lt;/span&gt; &lt;span class="dv"&gt;1&lt;/span&gt;
       &lt;span class="kw"&gt;ORDER&lt;/span&gt; &lt;span class="kw"&gt;BY&lt;/span&gt; &lt;span class="dv"&gt;2&lt;/span&gt; &lt;span class="kw"&gt;DESC&lt;/span&gt;
     ) a
&lt;span class="kw"&gt;LIMIT&lt;/span&gt; &lt;span class="dv"&gt;100&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Let's reproduce this piece by piece with Ibis. A few things first&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[19]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;investments&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;companies&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;clean_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;investor_name&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;NO INVESTOR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;investor_name&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;num_investments&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;permalink&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nunique&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;exited&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ipo&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;acquired&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ifelse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;permalink&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NA&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;num_exits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;exited&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nunique&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;These are just the basic statistics. Now let's join the tables and compute the acquisition rate:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[22]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;left_join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;permalink&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;company_permalink&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
         &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clean_name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
         &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;num_investments&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;num_investments&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                    &lt;span class="n"&gt;num_exits&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;num_exits&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;stats&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;acq_rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;num_exits&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; 
                                &lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;num_investments&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cast&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;float&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))))&lt;/span&gt;
&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[22]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
          investor_name  num_exits  num_investments  acq_rate
0                     .          0                1  0.000000
1         .406 Ventures          4               27  0.148148
2            01Startups          0                1  0.000000
3     1-800-FLOWERS.COM          0                1  0.000000
4             10 Others          0                1  0.000000
5          101 Startups          0                5  0.000000
6       105 Capital LLC          0                1  0.000000
7           10X Fund LP          0                1  0.000000
8  10X Venture Partners          0                2  0.000000
9           10Xelerator          1               21  0.047619
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Okay. Now let's look at the most successful investors who have made at least 100 investments:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[23]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;great_success&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;
                 &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;num_investments&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
                 &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;acq_rate&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="n"&gt;top20&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;great_success&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;top20&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[23]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
                     investor_name  num_exits  num_investments  acq_rate
0                       Ron Conway         52              119  0.436975
1                  Summit Partners         94              260  0.361538
2                 Trinity Ventures         42              125  0.336000
3                 Felicis Ventures         56              173  0.323699
4                  Austin Ventures         36              113  0.318584
5                   Menlo Ventures         54              171  0.315789
6                    Mayfield Fund         40              131  0.305344
7                              CRV         51              168  0.303571
8               Foundation Capital         41              137  0.299270
9                        Benchmark         52              180  0.288889
10  Norwest Venture Partners - NVP         41              144  0.284722
11                 Shasta Ventures         28              101  0.277228
12                SoftBank Capital         35              127  0.275591
13                        SV Angel        139              516  0.269380
14                    DAG Ventures         32              119  0.268908
15               Greylock Partners         62              231  0.268398
16               Ignition Partners         30              112  0.267857
17                         Venrock         40              153  0.261438
18                   True Ventures         39              152  0.256579
19                 Canaan Partners         37              146  0.253425
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Have fun with this data and look out for more blog posts on it to illustrate Ibis features.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Ibis 0.5: SQLite, Python 3, and more</title><link href="http://www.ibis-project.org/release-0.5/" rel="alternate"></link><updated>2015-09-09T08:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-09-09:release-0.5/</id><summary type="html">

&lt;p&gt;The next Ibis release is out, with some major new functionality:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;SQLite client and support for &lt;a href="https://docs.google.com/spreadsheets/d/11x29cmCBSj1V18g0kpxjoeooP9mTcJnQxsD1ikq9njg/edit?usp=sharing" target="_blank"&gt;most SQLite built-in functions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Python 3 compatibility (single codebase)&lt;/li&gt;
&lt;li&gt;SQLAlchemy-based expression translation toolchain to enable easier internal
  code reuse amongst SQL engines and pave road for PostgreSQL, Redshift,
  Vertica, and other analytic SQL engine support in the near future.&lt;/li&gt;
&lt;li&gt;Asynchronous query execution API (&lt;code&gt;expr.execute(async=True)&lt;/code&gt;) for Impala
  supporting query status and cancellation. This is very helpful in building
  multithreaded applications.&lt;/li&gt;
&lt;li&gt;Support for using Impala user-defined aggregate (UDA) functions&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There's a lot more, of course. Check out the detailed &lt;a href="http://docs.ibis-project.org/release.html" target="_blank"&gt;release notes&lt;/a&gt;, and read on for more about the upcoming roadmap.&lt;/p&gt;
&lt;p&gt;Check out &lt;a href="http://blog.ibis-project.org/sqlite-crunchbase-quickstart/" target="_blank"&gt;this follow-up post&lt;/a&gt; for a quick start in using
Ibis on SQLite with the &lt;a href="http://blog.ibis-project.org/pages/data.html"&gt;newly posted&lt;/a&gt; Crunchbase dataset.&lt;/p&gt;
&lt;p&gt;Install Ibis from PyPI with&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;pip install ibis-framework
&lt;/pre&gt;&lt;/div&gt;




&lt;p&gt;Thanks to all who contributed patches:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$ git log v0.4.0..v0.5.0 --pretty&lt;span class="o"&gt;=&lt;/span&gt;format:%aN &lt;span class="p"&gt;|&lt;/span&gt; sort &lt;span class="p"&gt;|&lt;/span&gt; uniq -c &lt;span class="p"&gt;|&lt;/span&gt; sort -rn
     &lt;span class="m"&gt;55&lt;/span&gt; Wes McKinney
      &lt;span class="m"&gt;9&lt;/span&gt; Uri Laserson
      &lt;span class="m"&gt;1&lt;/span&gt; Kristopher Overholt
&lt;/pre&gt;&lt;/div&gt;


&lt;h2&gt;Big news: expanding SQL engine support&lt;/h2&gt;
&lt;p&gt;One of the major goals of Ibis is to enable analytics work to be migrated from
SQL code to Python code. Since much data being warehoused in analytic SQL
systems (like Impala on HDFS or Redshift on AWS) isn't going anywhere soon,
architecturally this requires building a feature-complete SQL translation
toolchain. Ibis compiles Python to SQL behind the scenes and sends it to your
data engine of choice.&lt;/p&gt;
&lt;p&gt;We are taking SQL feature coverage very seriously. That means if you find a
&lt;code&gt;SELECT&lt;/code&gt; SQL query that cannot be expressed with Ibis, we will treat it as a
bug.&lt;/p&gt;
&lt;p&gt;Between Ibis 0.4 and 0.5, I undertook significant refactoring to separate
Impala-specific functionality from the more generic SQL compilation
toolchain. As part of this, I added a SQLAlchemy compiler-translator that
converts Ibis expressions into SQLAlchemy expressions. To see this through to
completion, I built a &lt;strong&gt;&lt;a href="http://blog.ibis-project.org/sqlite-crunchbase-quickstart/" target="_blank"&gt;SQLite Ibis client&lt;/a&gt;&lt;/strong&gt; that takes
advantage of this.&lt;/p&gt;
&lt;p&gt;Supporting more SQL engines is a lot of work, because each system has its own
set of built-in functions, and these have to be wrapped and connected to the
SQL-independent Ibis expression DSL.&lt;/p&gt;
&lt;p&gt;Having this flexible and reusable translation toolchain available also makes it
easier to smooth over behavior differences and API inconsistencies between SQL
engines.&lt;/p&gt;
&lt;p&gt;I would like to add more SQL engines; those designed for analytics (like
Redshift, Vertica, and Presto) are likely to receive more attention in the
short term. If you would like to get involved please get in touch.&lt;/p&gt;
&lt;h2&gt;Upcoming Ibis roadmap&lt;/h2&gt;
&lt;p&gt;Focus area in the coming months for the project will be:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Expanding SQL engine support (Redshift, Presto, Vertica, and Spark SQL are
  high priorities)&lt;/li&gt;
&lt;li&gt;Support for Impala complex (nested) types&lt;/li&gt;
&lt;li&gt;Tools for more complex ETL workflows on Impala&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;See the &lt;a href="https://github.com/cloudera/ibis/issues" target="_blank"&gt;GitHub issue tracker&lt;/a&gt; for the granular feature
roadmap.&lt;/p&gt;</summary></entry><entry><title>Ibis expressions: Pythonic composability and evaluation</title><link href="http://www.ibis-project.org/design-composability/" rel="alternate"></link><updated>2015-08-18T08:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-08-18:design-composability/</id><summary type="html">&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;

&lt;p&gt;The R community, especially my friend &lt;a href="http://twitter.com/hadleywickham"&gt;Hadley Wickham&lt;/a&gt;, has done pioneering work in creating expressive and useful domain-specific languages (DSLs) embedded in the R programming language. This articles gives some technical detail on composability and evaluation semantics in R and related work in Ibis to solve the same kinds of problems in a Pythonic way.&lt;/p&gt;


&lt;h1 id="non-standard-evaluation-in-dplyr"&gt;Non-standard evaluation in dplyr&lt;/h1&gt;
&lt;p&gt;In Hadley's book &lt;em&gt;Advanced R&lt;/em&gt;, he gives &lt;a href="http://adv-r.had.co.nz/Computing-on-the-language.html"&gt;an extensive look&lt;/a&gt; at a tool available to R programmers called &lt;em&gt;non-standard evaluation&lt;/em&gt; (NSE). Users of &lt;code&gt;ggplot2&lt;/code&gt; or &lt;code&gt;dplyr&lt;/code&gt; (or other Hadleyverse packages) will be familiar with this — it is also used in base R for model formulas.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;NB. For the programming language junkies, NSE originates from the homoiconic Lisp tradition of &amp;quot;code as data&amp;quot;. The R language implementation &lt;em&gt;is&lt;/em&gt; at its core a kind of Lisp interpreter!&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Let's take an example from one of the dplyr vignettes on everyone's favorite &lt;a href="http://blog.ibis-project.org/pages/data.html"&gt;airlines dataset&lt;/a&gt;.&lt;/p&gt;
&lt;pre class="sourceCode R"&gt;&lt;code class="sourceCode r"&gt;airlines %&amp;gt;%
&lt;span class="st"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;filter&lt;/span&gt;(!&lt;span class="kw"&gt;is.na&lt;/span&gt;(dep_delay)) %&amp;gt;%
&lt;span class="st"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;group_by&lt;/span&gt;(year, month, day) %&amp;gt;%
&lt;span class="st"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;summarise&lt;/span&gt;(&lt;span class="dt"&gt;delay =&lt;/span&gt; &lt;span class="kw"&gt;mean&lt;/span&gt;(dep_delay)) %&amp;gt;%
&lt;span class="st"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;arrange&lt;/span&gt;(&lt;span class="kw"&gt;desc&lt;/span&gt;(delay)) %&amp;gt;%
&lt;span class="st"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;head&lt;/span&gt;(&lt;span class="dv"&gt;5&lt;/span&gt;)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As a Python programmer, you might ask &amp;quot;what in tarnation is going on here?&amp;quot;. Indeed, very few of the symbols in this code block are valid variables in the local scope. The short story is that the code (or &lt;em&gt;data&lt;/em&gt; would be more accurate) inside the function calls is evaluated in the context of the &amp;quot;calling&amp;quot; data frame. So the NSE code&lt;/p&gt;
&lt;pre class="sourceCode R"&gt;&lt;code class="sourceCode r"&gt;airlines %&amp;gt;%&lt;span class="st"&gt; &lt;/span&gt;
&lt;span class="st"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;filter&lt;/span&gt;(!&lt;span class="kw"&gt;is.na&lt;/span&gt;(dep_delay))&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Is equivalent to the more explicit non-NSE code (written here without the pipe &lt;code&gt;%&amp;gt;%&lt;/code&gt;):&lt;/p&gt;
&lt;pre class="sourceCode R"&gt;&lt;code class="sourceCode r"&gt;&lt;span class="kw"&gt;filter&lt;/span&gt;(airlines, !&lt;span class="kw"&gt;is.na&lt;/span&gt;(airlines$dep_delay))&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;What is really nice about NSE is that you can refer to new data frame columns that appear during a set of composed operations. For example, the step &lt;code&gt;arrange(desc(delay))&lt;/code&gt; refers to &lt;code&gt;delay&lt;/code&gt; which appeared during the aggregation &lt;code&gt;summarise(delay = mean(dep_delay))&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The sad thing about NSE is that it's &lt;em&gt;very&lt;/em&gt; unpythonic. It goes directly against a core tenet: &amp;quot;Explicit is better than implicit.&amp;quot; It also can make debugging more difficult with complex expressions involving many NSE variables and functions.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Doing NSE in Python actually &lt;em&gt;is&lt;/em&gt; possible, but it won't make you very many friends. I wrote a &lt;a href="http://wesmckinney.com/blog/python-r-and-the-allure-of-magic/"&gt;blog post about it&lt;/a&gt; several years ago.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;In the above example, the lack of NSE doesn't hurt us very much:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;
&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactive&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bottou01.sjc.cloudera.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;wes.airlines_parquet&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;depdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notnull&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;month&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;day&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;depdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[1]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   year  month  day      delay
0  2001      9   13  75.139792
1  2001      9   14  47.194577
2  1990     12   21  45.661782
3  1990     12   22  45.222249
4  1990     12   28  43.914432
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Okay, so we're able to do the same thing using strings to reference columns, but let's take a more complex example where this will fail us and require writing down local variables. The trouble comes from needing to form some non-trivial expression during a set of composed operations:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;depdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notnull&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;leg&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;literal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;-&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;month&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;day&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;depdelay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;leg&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[2]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   year  month  day  depdelay      leg
0  1999     12   22        -1  SDF-CVG
1  1999     12   23         2  SDF-CVG
2  1999     12   24         3  SDF-CVG
3  1999     12   25         0  SDF-CVG
4  1999     12   26         3  SDF-CVG
5  1999     12   27         2  SDF-CVG
6  1999     12   28         0  SDF-CVG
7  1999     12   29        -4  SDF-CVG
8  1999     12   30        -8  SDF-CVG
9  1999     12   31        -2  SDF-CVG
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Suppose now we wanted to only look at a few legs of interest, like &lt;code&gt;SFO-JFK&lt;/code&gt; and &lt;code&gt;SFO-EWR&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;leg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;SFO-JFK&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;SFO-EWR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[3]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   year  month  day  depdelay      leg
0  1995     12    1         0  SFO-JFK
1  1995     12    2        37  SFO-JFK
2  1995     12    3         1  SFO-JFK
3  1995     12    4         4  SFO-JFK
4  1995     12    5         0  SFO-JFK
5  1995     12    6         1  SFO-JFK
6  1995     12    7         0  SFO-JFK
7  1995     12    8         3  SFO-JFK
8  1995     12   10         2  SFO-JFK
9  1995     12   11        31  SFO-JFK
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Hmm.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h1 id="late-binding-using-functions"&gt;&amp;quot;Late&amp;quot; binding using functions&lt;/h1&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;One Pythonic solution to the composability conundrum is to pass functions as parameters. This simulates NSE's &amp;quot;late binding&amp;quot; semantics in a way that doesn't break any rules about standard Python best practices.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;depdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notnull&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;leg&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;literal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;-&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;leg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;SFO-JFK&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;SFO-EWR&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;month&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;leg&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;depdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                    &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;# flights&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   year  month      leg  avg_delay  # flights
0  2008     12  SFO-EWR  37.526316        247
1  1987     12  SFO-EWR  34.297030        101
2  2008      6  SFO-EWR  31.103203        281
3  2008      1  SFO-EWR  29.906780        236
4  2008      6  SFO-JFK  29.676998        613
5  1988      1  SFO-EWR  28.398058        103
6  2008      3  SFO-EWR  24.672269        238
7  2000      5  SFO-EWR  23.857923        366
8  2008      7  SFO-JFK  23.512821        624
9  1989      3  SFO-EWR  23.114035        114
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;It's a bit of a mouthful with those extra &lt;code&gt;lambda&lt;/code&gt;'s, but it's also kind of nice! Note that Ibis doesn't force you to write one long set of composed operations (feel free to write down as many local variables saving intermediate expressions as you like), but it's nice to have the option! (Also, remind me to avoid Newark airport.)&lt;/p&gt;
&lt;p&gt;For the curious, here's the SQL that's generated by this expression:&lt;/p&gt;
&lt;pre class="sourceCode sql"&gt;&lt;code class="sourceCode sql"&gt;&lt;span class="kw"&gt;SELECT&lt;/span&gt; t0.`year`, t0.`month`, t0.`leg`, &lt;span class="fu"&gt;avg&lt;/span&gt;(t0.`depdelay`) &lt;span class="kw"&gt;AS&lt;/span&gt; `avg_delay`,
       &lt;span class="fu"&gt;count&lt;/span&gt;(*) &lt;span class="kw"&gt;AS&lt;/span&gt; `# flights`
&lt;span class="kw"&gt;FROM&lt;/span&gt; (
  &lt;span class="kw"&gt;SELECT&lt;/span&gt; *, concat_ws(&lt;span class="st"&gt;&amp;#39;-&amp;#39;&lt;/span&gt;, `origin`, `dest`) &lt;span class="kw"&gt;AS&lt;/span&gt; `leg`
  &lt;span class="kw"&gt;FROM&lt;/span&gt; wes.airlines_parquet
  &lt;span class="kw"&gt;WHERE&lt;/span&gt; `depdelay` &lt;span class="kw"&gt;IS&lt;/span&gt; &lt;span class="kw"&gt;NOT&lt;/span&gt; &lt;span class="kw"&gt;NULL&lt;/span&gt;
) t0
&lt;span class="kw"&gt;WHERE&lt;/span&gt; t0.`leg` &lt;span class="kw"&gt;IN&lt;/span&gt; (&lt;span class="st"&gt;&amp;#39;SFO-JFK&amp;#39;&lt;/span&gt;, &lt;span class="st"&gt;&amp;#39;SFO-EWR&amp;#39;&lt;/span&gt;)
&lt;span class="kw"&gt;GROUP&lt;/span&gt; &lt;span class="kw"&gt;BY&lt;/span&gt; &lt;span class="dv"&gt;1&lt;/span&gt;, &lt;span class="dv"&gt;2&lt;/span&gt;, &lt;span class="dv"&gt;3&lt;/span&gt;
&lt;span class="kw"&gt;ORDER&lt;/span&gt; &lt;span class="kw"&gt;BY&lt;/span&gt; `avg_delay` &lt;span class="kw"&gt;DESC&lt;/span&gt;
&lt;span class="kw"&gt;LIMIT&lt;/span&gt; &lt;span class="dv"&gt;10&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h1 id="the-pipe-method"&gt;The &lt;code&gt;pipe&lt;/code&gt; method&lt;/h1&gt;
&lt;p&gt;Another tool to aid composability is the &lt;code&gt;pipe&lt;/code&gt; table method, which was also added to pandas in version 0.16.2. Basically, &lt;code&gt;pipe&lt;/code&gt; makes the statement&lt;/p&gt;
&lt;pre class="sourceCode python"&gt;&lt;code class="sourceCode python"&gt;table.pipe(f, **f_kwargs).pipe(g, **g_kwargs)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;equivalent to&lt;/p&gt;
&lt;pre class="sourceCode python"&gt;&lt;code class="sourceCode python"&gt;g(f(table, **f_kwargs), **g_kwargs)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In R, you could write:&lt;/p&gt;
&lt;pre class="sourceCode R"&gt;&lt;code class="sourceCode r"&gt;table %&amp;gt;%&lt;span class="st"&gt; &lt;/span&gt;&lt;span class="kw"&gt;f&lt;/span&gt;(...) %&amp;gt;%&lt;span class="st"&gt; &lt;/span&gt;&lt;span class="kw"&gt;g&lt;/span&gt;(...)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So in Ibis, you could write little helper functions that deal in common table idioms for your application. Let me give you an example that removes any rows containing nulls amongst an input set of columns:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;remove_nulls_in&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;f&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;mask&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notnull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mask&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;mask&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cond&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;

&lt;span class="n"&gt;filtered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;remove_nulls_in&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;depdelay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;arrdelay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Using these tools, you can augment the built-in Ibis DSL constructs to suit your needs, implementing analytics faster with less code.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Leveraging SQL window functions in Ibis</title><link href="http://www.ibis-project.org/sql-window-functions/" rel="alternate"></link><updated>2015-08-17T08:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-08-17:sql-window-functions/</id><summary type="html">&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;


&lt;p&gt;&lt;em&gt;Window&lt;/em&gt; (also known as &lt;em&gt;analytic&lt;/em&gt;) functions are a valuable technique in analytic SQL, but unfortunately they are generally considered to be an &lt;em&gt;advanced&lt;/em&gt; skill among SQL programmers. Conceptually, they are relatively simple, and indeed many everyday pandas and R operations can be expressed in SQL through their use. Mechanically, they can be difficult to use, largely because of the SQL syntax.&lt;/p&gt;
&lt;p&gt;Ibis has had comprehensive support for window functions from 0.3 onward, and I invested quite a bit of effort to design an API to make them available to users in a much simpler way. I also made sure that you don't have to be a SQL expert to use them. &lt;/p&gt;
&lt;h1 id="what-are-sql-window-functions"&gt;What are SQL window functions?&lt;/h1&gt;
&lt;p&gt;Standard SQL functions do not have &amp;quot;data visibility&amp;quot; outside the context of the current row. This makes group-wise data analysis more difficult. In pandas, you can write code that looks like:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;demeaned_col = df.col_name - df.col_name.mean()

first_order_diff = df.col_name - df.col_name.shift(1)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;But you can't write this SQL&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;SELECT col_name - AVG(col_name) AS demeaned_col
FROM df&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is where window functions come in. They give you a way to group and order an entire data set and compute either an &lt;em&gt;aggregation&lt;/em&gt; or some other &lt;em&gt;manipulation&lt;/em&gt; of ordered groups. Depending on your SQL engine, you probably have functions like:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Aggregations: standard aggregates like &lt;code&gt;AVG&lt;/code&gt; and &lt;code&gt;SUM&lt;/code&gt; can be used in window function contexts&lt;/li&gt;
&lt;li&gt;Shifts: &lt;code&gt;LAG&lt;/code&gt; and &lt;code&gt;LEAD&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Quantiling: &lt;code&gt;NTILE&lt;/code&gt;, and others&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These collectively are called &lt;strong&gt;analytic functions&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Mechanically, a SQL window function involves adorning the &lt;code&gt;OVER&lt;/code&gt; keyword along with 3 components:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Group clause: &lt;code&gt;PARTITION BY&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Sort clause: &lt;code&gt;ORDER BY&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Window frame: it's complicated&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Without going into great detail about all the ins and outs of SQL &lt;code&gt;OVER&lt;/code&gt; clauses, here's an example of 1 year deviation from the full sample average of a value on some monthly data. In pandas, this would be:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;def apply_func(g):
    g = g.sort(&amp;#39;date&amp;#39;)
    metric = pd.rolling_mean(g.value - df.value.mean(), 12)
    return pd.DataFrame({&amp;#39;metric&amp;#39;: metric, &amp;#39;state&amp;#39;: g.state})

result = df.groupby(&amp;#39;state&amp;#39;).apply(apply_func)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In SQL, here it is:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;SELECT AVG(value - AVG(value) OVER ()) 
       OVER(PARTITION BY state
            ORDER BY date
            RANGE BETWEEN 11 PRECEDING AND CURRENT ROW)
       AS metric
FROM df&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I feel like we can do better than this.&lt;/p&gt;
&lt;h1 id="more-pain-free-group-wise-analytics-in-ibis"&gt;More pain-free group-wise analytics in Ibis&lt;/h1&gt;
&lt;p&gt;In many cases, window functions are only an implementation detail. In Ibis, I've tried to make using them as pain-free as possible, and invisible in a lot of cases. There's still &lt;a href="https://github.com/cloudera/ibis/issues?utf8=%E2%9C%93&amp;amp;q=is%3Aissue+is%3Aopen+window"&gt;plenty more to do&lt;/a&gt; and more usability improvements and higher level tools (getting around SQL limitations) will appear in upcoming releases.&lt;/p&gt;
&lt;p&gt;Let's look at the &lt;a href="http://blog.ibis-project.org/pages/data.html"&gt;airlines data&lt;/a&gt; again.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;
&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactive&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;

&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bottou01.sjc.cloudera.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;db&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;wes&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;airlines_parquet&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Suppose you wanted to demean the &lt;code&gt;arrdelay&lt;/code&gt; column in this dataset. Here you can write:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[2]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
0   -14.049963
1   -11.049963
2    -0.049963
3   -13.049963
4    -5.049963
5   -12.049963
6    -0.049963
7   -13.049963
8   -19.049963
9   -10.049963
Name: tmp, dtype: float64
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's take a closer look:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; 
         &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;demeaned&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[3]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   arrdelay      mean   demeaned
0        -7  7.049963 -14.049963
1        -4  7.049963 -11.049963
2         7  7.049963  -0.049963
3        -6  7.049963 -13.049963
4         2  7.049963  -5.049963
5        -5  7.049963 -12.049963
6         7  7.049963  -0.049963
7        -6  7.049963 -13.049963
8       -12  7.049963 -19.049963
9        -3  7.049963 -10.049963
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;How about if you wanted to demean by &lt;code&gt;dest&lt;/code&gt;. Now you can write:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;arrdelay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dest_avg&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
                &lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;

&lt;span class="n"&gt;worst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notnull&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dev&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;worst&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   arrdelay dest  dest_avg          dev
0      2598  DTW  4.998577  2593.001423
1      2461  MSP  4.476601  2456.523399
2      2453  MSP  4.476601  2448.523399
3      2137  MSP  4.476601  2132.523399
4      1951  DTW  4.998577  1946.001423
5      1946  DTW  4.998577  1941.001423
6      1942  MSP  4.476601  1937.523399
7      1925  SEA  8.158243  1916.841757
8      1879  SEA  8.158243  1870.841757
9      1864  MSP  4.476601  1859.523399
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;For the SQL junkies in the room, here was the actual query that Ibis executed&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;SELECT *
FROM (
  SELECT `arrdelay`, `dest`,
         avg(`arrdelay`) OVER (PARTITION BY `dest`) AS `dest_avg`,
         `arrdelay` - avg(`arrdelay`) OVER (PARTITION BY `dest`) AS `dev`
  FROM wes.`airlines_parquet`
) t0
WHERE `dev` IS NOT NULL
ORDER BY `dev` DESC
LIMIT 10&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Note that you can explicitly construct a window to use with analytic functions and use the &lt;code&gt;over&lt;/code&gt; expression method:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;

&lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;
&lt;span class="n"&gt;yoy_chg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lag&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;over&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;yoy_change&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yoy_chg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;ORD&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                 &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;yearly_avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[5]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
  dest  year  avg_delay  yoy_change
0  ORD  2000  16.630993    4.742508
1  ORD  2001   7.849993   -8.781000
2  ORD  2002   3.597082   -4.252911
3  ORD  2003   7.268648    3.671566
4  ORD  2004  12.812650    5.544002
5  ORD  2005   8.563716   -4.248934
6  ORD  2006  15.744943    7.181227
7  ORD  2007  16.804941    1.059998
8  ORD  2008  15.224999   -1.579942
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h1 id="moving-window-functions-and-ordering"&gt;Moving window functions and ordering&lt;/h1&gt;
&lt;p&gt;Aggregations (like &lt;code&gt;sum&lt;/code&gt;, &lt;code&gt;mean&lt;/code&gt;, etc.) used in a window function context by default operate on the full window data sample. This is configurable, though, and you can choose which values to include in the aggregated sample based on a set ordering. Let's take the annual delays from above for a couple airports:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[6]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;JFK&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;SFO&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;                
                &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
                &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[6]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   dest  year  avg_delay
0   JFK  1993   6.441211
1   JFK  2002   2.723808
2   JFK  1999   7.005718
3   SFO  2003   3.977823
4   SFO  1998  17.903815
5   JFK  1994   9.017850
6   SFO  1999  12.464576
7   JFK  2001   8.875575
8   SFO  1990   8.622028
9   JFK  1998   5.931865
10  JFK  2003   2.930521
11  SFO  1995  10.024748
12  SFO  2006  12.377844
13  JFK  2007  18.385748
14  JFK  1987   9.168451
15  JFK  1989   9.387740
16  JFK  2008  13.774854
17  JFK  2000   9.554254
18  JFK  2004   6.992708
19  SFO  2001   8.233889
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;If you wanted to add a full sample average-of-averages by each &lt;code&gt;dest&lt;/code&gt;, you could write this code&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[7]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;enriched&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;
            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;grand_avg&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="n"&gt;enriched&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[7]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;dest&lt;/th&gt;
      &lt;th&gt;year&lt;/th&gt;
      &lt;th&gt;avg_delay&lt;/th&gt;
      &lt;th&gt;grand_avg&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;1991&lt;/td&gt;
      &lt;td&gt;9.211260&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;2004&lt;/td&gt;
      &lt;td&gt;6.034957&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;1993&lt;/td&gt;
      &lt;td&gt;6.788378&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;2002&lt;/td&gt;
      &lt;td&gt;4.721505&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;1994&lt;/td&gt;
      &lt;td&gt;6.266924&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;5&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;1992&lt;/td&gt;
      &lt;td&gt;4.347605&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;6&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;2005&lt;/td&gt;
      &lt;td&gt;9.145214&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;7&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;1988&lt;/td&gt;
      &lt;td&gt;8.897241&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;8&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;19.235428&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;9&lt;/th&gt;
      &lt;td&gt;SFO&lt;/td&gt;
      &lt;td&gt;1997&lt;/td&gt;
      &lt;td&gt;9.026597&lt;/td&gt;
      &lt;td&gt;10.58818&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Suppose now you wanted to compare with trailing 10 year average instead. Ibis gives you two ways to do that, the generic &lt;code&gt;ibis.window&lt;/code&gt; or convenience method &lt;code&gt;ibis.trailing_window&lt;/code&gt;:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[8]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# 9 + current period = 10&lt;/span&gt;
&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                &lt;span class="n"&gt;order_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;preceding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;following&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;w2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trailing_window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                          &lt;span class="n"&gt;order_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Now, you can use the &lt;code&gt;over&lt;/code&gt; method:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[9]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;over&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;enriched&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;enriched&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[9]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
  dest  year  avg_delay       mean
0  SFO  2008  13.651567  10.233152
1  JFK  2008  13.774854   9.566769
2  JFK  2007  18.385748   8.782470
3  SFO  2007  12.488720  10.658377
4  SFO  2006  12.377844  10.312165
5  JFK  2006  13.109659   7.560915
6  JFK  2005  12.314841   7.784602
7  SFO  2005   9.145214  10.572521
8  SFO  2004   6.034957  10.660475
9  JFK  2004   6.992708   7.443993
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;I could also have achieved the same result by doing:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[10]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;w3&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trailing_window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;avg_10y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;annual_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;over&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w3&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[10]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
  dest  year  avg_delay    avg_10y
0  SFO  2008  13.651567  10.233152
1  JFK  2008  13.774854   9.566769
2  JFK  2007  18.385748   9.236323
3  SFO  2007  12.488720  10.930485
4  SFO  2006  12.377844  10.771828
5  JFK  2006  13.109659   8.980813
6  JFK  2005  12.314841   9.470483
7  SFO  2005   9.145214  11.095642
8  SFO  2004   6.034957  11.019149
9  JFK  2004   6.992708   9.430359
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Some functions, like any of the &lt;code&gt;cum*&lt;/code&gt; methods, like &lt;code&gt;cumsum&lt;/code&gt;, automatically turn into the right window functions under the hood.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h1 id="all-the-window-frames-with-ibis"&gt;All the window frames with Ibis&lt;/h1&gt;
&lt;p&gt;I made a handy illustration (hat tip to &lt;a href="https://twitter.com/hadleywickham"&gt;Hadley Wickham&lt;/a&gt; and his &lt;a href="https://rpubs.com/hadley/window"&gt;post on dplyr&lt;/a&gt;) illustrating how to create each kind of window with Ibis:&lt;/p&gt;
&lt;center&gt;
&lt;img src="../images/window-frame.png"&gt;&lt;/img&gt;
&lt;/center&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h1 id="helping-more-window-composability"&gt;Helping more: window composability&lt;/h1&gt;
&lt;p&gt;You can think of a &lt;code&gt;Window&lt;/code&gt; object as providing a description of how to group a data set. So why not make the windows &lt;em&gt;composable&lt;/em&gt;? Let's consider a case where you have two different windows is an analysis:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[12]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;w1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;month&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="n"&gt;w2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trailing_window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;To simplify making a window that's the union of all these parameters, we can simply do:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[13]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;combine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;You can even construct the combined window yourself and check that they're the same:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[14]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;w3&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order_by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;year&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;month&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                 &lt;span class="n"&gt;preceding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;following&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;equals&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[14]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
True
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h1 id="takeaway"&gt;Takeaway&lt;/h1&gt;
&lt;p&gt;There's no reason for analytics in a SQL engine requiring window functions to be so tedious. If you're an avid user of the tools described in this post, I'd love to hear your feedback about how to make things even more productive.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Ibis 0.4.0 release</title><link href="http://www.ibis-project.org/release-0.4/" rel="alternate"></link><updated>2015-08-14T09:30:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-08-14:release-0.4/</id><summary type="html">

&lt;p&gt;The next Ibis release is out, check out the detailed &lt;a href="http://docs.ibis-project.org/release.html"&gt;release notes&lt;/a&gt;.&lt;/p&gt;


&lt;p&gt;Notable features in this release (some of which will be the subject of upcoming
blog posts):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Using Impala user-defined C++ functions with Ibis (&lt;a href="http://docs.ibis-project.org/impala-udf.html"&gt;docs here&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Better composability and chaining by passing functions as parameters&lt;/li&gt;
&lt;li&gt;An ORM-like layer for interacting with database objects&lt;/li&gt;
&lt;li&gt;&lt;a href="http://blog.ibis-project.org/design-modeling-analytics-1/"&gt;Enhanced TopK expressions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Support for Kerberos-enabled secure HDFS clusters&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Thanks to all who contributed patches:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$ git log v0.3.0..v0.4.0 --pretty&lt;span class="o"&gt;=&lt;/span&gt;format:%aN &lt;span class="p"&gt;|&lt;/span&gt; sort &lt;span class="p"&gt;|&lt;/span&gt; uniq -c &lt;span class="p"&gt;|&lt;/span&gt; sort -rn
     &lt;span class="m"&gt;38&lt;/span&gt; Wes McKinney
      &lt;span class="m"&gt;9&lt;/span&gt; Uri Laserson
      &lt;span class="m"&gt;2&lt;/span&gt; Meghana Vuyyuru
      &lt;span class="m"&gt;2&lt;/span&gt; Kristopher Overholt
      &lt;span class="m"&gt;1&lt;/span&gt; Marius van Niekerk
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The next major release will offer&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Python 3 support (this was blocked on upstream dependencies, sorry)&lt;/li&gt;
&lt;li&gt;Impala C++ aggregate function (UDA) support&lt;/li&gt;
&lt;li&gt;Async expression execution API (cancellation, progress, etc.)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I'm looking for people interested in helping build Ibis support for any of the
following compute engines:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Hive&lt;/li&gt;
&lt;li&gt;PostgreSQL flavors: Redshift, PostgreSQL, HP Vertica&lt;/li&gt;
&lt;li&gt;MySQL&lt;/li&gt;
&lt;li&gt;SQLite&lt;/li&gt;
&lt;li&gt;Presto&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If interested please contact me.&lt;/p&gt;</summary></entry><entry><title>Ibis Design: Modeling high level analytics tasks</title><link href="http://www.ibis-project.org/design-modeling-analytics-1/" rel="alternate"></link><updated>2015-08-10T08:00:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-08-10:design-modeling-analytics-1/</id><summary type="html">&lt;p&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;

&lt;p&gt;Outside of scalability and high performance on large data sets with Python, Ibis is focused on simplifying analytics tasks for end users. By designing a rich pandas-like domain specific language (DSL) embedded in Python code, we can hide away the complexities normally associated with expressing analytical concepts in SQL or some other tool. This post gives some specific examples and shows how we're solving them in Ibis. &lt;/p&gt;
&lt;!-- 
* [TopK case study](#topkcasestudy)
* [A better TopK](#bettertopk)
* [Another example: bucketing](#bucketing)
--&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="case-study-the-topk-operation"&gt;&lt;a href="topkcasestudy"&gt;&lt;/a&gt; Case Study: The TopK Operation&lt;/h2&gt;
&lt;p&gt;We've all been there; you have some higher cardinality category and you wish to restrict your analysis to a fixed subset of values according to a ranking that you devise. If you're using pandas, you might do something like:&lt;/p&gt;
&lt;pre class="sourceCode python"&gt;&lt;code class="sourceCode python"&gt;K = &lt;span class="dv"&gt;5&lt;/span&gt;

avg_value = data.groupby(&lt;span class="st"&gt;&amp;#39;category&amp;#39;&lt;/span&gt;).value.mean()
top_categories = avg_value.order(ascending=&lt;span class="ot"&gt;False&lt;/span&gt;)[:K].index

filtered_data = data[data.category.isin(top_categories)]&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Let's look at a concrete example using both Ibis and SQL. I'm using the now-classic &lt;a href="http://blog.ibis-project.org/pages/data.html"&gt;airlines dataset&lt;/a&gt; in Parquet format.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[1]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;ibis&lt;/span&gt;

&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interactive&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;

&lt;span class="n"&gt;hdfs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hdfs_connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bottou01.sjc.cloudera.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;impala&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bottou01.sjc.cloudera.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hdfs_client&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hdfs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;airlines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parquet_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/user/wesm/airlines-parquet&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                            &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;airlines_parquet&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;database&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;wes&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# airlines.compute_stats()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Let's look at the top 5 origin airports by total number of flights:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[2]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="s2"&gt;SELECT origin, count(*) as `nrows`&lt;/span&gt;
&lt;span class="s2"&gt;FROM wes.airlines_parquet&lt;/span&gt;
&lt;span class="s2"&gt;GROUP BY 1&lt;/span&gt;
&lt;span class="s2"&gt;ORDER BY nrows DESC&lt;/span&gt;
&lt;span class="s2"&gt;LIMIT 5&lt;/span&gt;
&lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[2]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
  origin    nrows
0    ORD  6597442
1    ATL  6100953
2    DFW  5710980
3    LAX  4089012
4    PHX  3491077
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;Constructed from SQL primitives, the TopK operation consists of 3 tasks&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Aggregate&lt;/li&gt;
&lt;li&gt;Sort&lt;/li&gt;
&lt;li&gt;Limit&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here is the Impala query plan for this (use &lt;code&gt;con.explain(expr)&lt;/code&gt; with Ibis to see it)&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;05:MERGING-EXCHANGE [UNPARTITIONED]
|  order by: count(*) DESC
|  limit: 5
|
02:TOP-N [LIMIT=5]
|  order by: count(*) DESC
|
04:AGGREGATE [FINALIZE]
|  output: count:merge(*)
|  group by: origin
|
03:EXCHANGE [HASH(origin)]
|
01:AGGREGATE
|  output: count(*)
|  group by: origin
|
00:SCAN HDFS [wes.airlines_parquet]
   partitions=1/1 files=8 size=1.34GB&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As you can see, Impala, like most SQL engines, has the built-in notion of &lt;code&gt;TOP-N&lt;/code&gt; in its query engine, but yet this is not expressible directly in SQL. Things get more complicated when you want to filter down to the categories in the result:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;SELECT dest, avg(arrdelay) as `nrows`
FROM wes.airlines_parquet
WHERE origin in (
  SELECT origin 
  FROM (
    SELECT origin, count(*) as `nrows`
    FROM wes.airlines_parquet
    GROUP BY 1
    ORDER BY nrows DESC
    LIMIT 5  
  ) t0
)
GROUP BY 1&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If you look at the query plan for this, it's more complicated but functionally contains the same TopK pattern plus a filter (using a &lt;code&gt;LEFT SEMI JOIN&lt;/code&gt;, a type of efficient &lt;em&gt;filter join&lt;/em&gt;).&lt;/p&gt;
&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="a-better-topk-workflow-with-ibis"&gt;&lt;a href="bettertopk"&gt;&lt;/a&gt; A better TopK workflow with Ibis&lt;/h2&gt;
&lt;p&gt;Ibis is happy to let you build your own TopK expression using primitive operations:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[3]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;
               &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;origin&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
               &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;nrows&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
               &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;nrows&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
               &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;top_origins&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[3]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;origin&lt;/th&gt;
      &lt;th&gt;nrows&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;ORD&lt;/td&gt;
      &lt;td&gt;6597442&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;ATL&lt;/td&gt;
      &lt;td&gt;6100953&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;DFW&lt;/td&gt;
      &lt;td&gt;5710980&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;LAX&lt;/td&gt;
      &lt;td&gt;4089012&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;PHX&lt;/td&gt;
      &lt;td&gt;3491077&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This same aggregated table can be used as a filter, too:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[4]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# Replace nulls with 0&lt;/span&gt;
&lt;span class="n"&gt;avg_delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;subset_airlines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
                    
&lt;span class="n"&gt;avg_delay_subset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;subset_airlines&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
                    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;avg_delay_subset&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[4]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;dest&lt;/th&gt;
      &lt;th&gt;avg_delay&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;RDM&lt;/td&gt;
      &lt;td&gt;29.558824&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;SOP&lt;/td&gt;
      &lt;td&gt;25.598706&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;MQT&lt;/td&gt;
      &lt;td&gt;25.367129&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;HHH&lt;/td&gt;
      &lt;td&gt;23.744963&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;ACY&lt;/td&gt;
      &lt;td&gt;20.344538&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;5&lt;/th&gt;
      &lt;td&gt;PSE&lt;/td&gt;
      &lt;td&gt;19.000000&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;6&lt;/th&gt;
      &lt;td&gt;CPR&lt;/td&gt;
      &lt;td&gt;18.383459&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;7&lt;/th&gt;
      &lt;td&gt;CWA&lt;/td&gt;
      &lt;td&gt;18.002533&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;8&lt;/th&gt;
      &lt;td&gt;SCE&lt;/td&gt;
      &lt;td&gt;16.741325&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;9&lt;/th&gt;
      &lt;td&gt;ATW&lt;/td&gt;
      &lt;td&gt;16.138655&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;This is quite a bit of data gymnastics, though, and you haven't really modeled the fundamental Top-N task with the API. Because we have the power to create higher-level abstractions and expand them later into concrete operations (backend-dependent), we can craft a &lt;code&gt;topk&lt;/code&gt; operation:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[5]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;The result of &lt;code&gt;topk&lt;/code&gt; in Ibis is what's known as an &lt;em&gt;analytic expression&lt;/em&gt;. Its behavior depends on the context in which it's used. It is executable as is, yielding the top 5 origins by count:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[6]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[6]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;origin&lt;/th&gt;
      &lt;th&gt;count&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;ORD&lt;/td&gt;
      &lt;td&gt;6597442&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;ATL&lt;/td&gt;
      &lt;td&gt;6100953&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;DFW&lt;/td&gt;
      &lt;td&gt;5710980&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;LAX&lt;/td&gt;
      &lt;td&gt;4089012&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;PHX&lt;/td&gt;
      &lt;td&gt;3491077&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;But you can also use it directly as a table filter:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[7]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dest&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ibis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[7]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;dest&lt;/th&gt;
      &lt;th&gt;avg_delay&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;RDM&lt;/td&gt;
      &lt;td&gt;29.558824&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;SOP&lt;/td&gt;
      &lt;td&gt;25.598706&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;MQT&lt;/td&gt;
      &lt;td&gt;25.367129&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;HHH&lt;/td&gt;
      &lt;td&gt;23.744963&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;ACY&lt;/td&gt;
      &lt;td&gt;20.344538&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;5&lt;/th&gt;
      &lt;td&gt;PSE&lt;/td&gt;
      &lt;td&gt;19.000000&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;6&lt;/th&gt;
      &lt;td&gt;CPR&lt;/td&gt;
      &lt;td&gt;18.383459&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;7&lt;/th&gt;
      &lt;td&gt;CWA&lt;/td&gt;
      &lt;td&gt;18.002533&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;8&lt;/th&gt;
      &lt;td&gt;SCE&lt;/td&gt;
      &lt;td&gt;16.741325&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;9&lt;/th&gt;
      &lt;td&gt;ATW&lt;/td&gt;
      &lt;td&gt;16.138655&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;What's going on here is that the &lt;code&gt;top_origins&lt;/code&gt; expression is a sort of &lt;em&gt;logical data type&lt;/em&gt; that hasn't yet been adapted to any physical data operations. There's a number of ways it can be used, and that adaptation process happens automatically:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[8]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[8]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
ibis.expr.operations.TopKExpr
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;You might notice that I've sneakily done two TopK operations in a row, the first ranking by count and the second by average arrival delay. &lt;code&gt;topk&lt;/code&gt; luckily gives you a way to provide a ranking metric other than count:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[9]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[9]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;origin&lt;/th&gt;
      &lt;th&gt;avg_delay&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;FMN&lt;/td&gt;
      &lt;td&gt;164.0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;OGD&lt;/td&gt;
      &lt;td&gt;148.4&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;CYS&lt;/td&gt;
      &lt;td&gt;122.0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;BFF&lt;/td&gt;
      &lt;td&gt;93.0&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;PIR&lt;/td&gt;
      &lt;td&gt;33.5&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;We can combine all this together to do a double-TopN:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[10]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;avg_delay&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;avg_delay&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    
&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;top_origins&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;avg_delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[10]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;

&lt;div class="output_html rendered_html"&gt;
&lt;div style="max-width:1500px;overflow:auto;"&gt;
&lt;table border="1" class="dataframe"&gt;
  &lt;thead&gt;
    &lt;tr style="text-align: right;"&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;dest&lt;/th&gt;
      &lt;th&gt;avg_delay&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;th&gt;0&lt;/th&gt;
      &lt;td&gt;RDM&lt;/td&gt;
      &lt;td&gt;29.558824&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;1&lt;/th&gt;
      &lt;td&gt;SOP&lt;/td&gt;
      &lt;td&gt;25.598706&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;2&lt;/th&gt;
      &lt;td&gt;MQT&lt;/td&gt;
      &lt;td&gt;25.367129&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;3&lt;/th&gt;
      &lt;td&gt;HHH&lt;/td&gt;
      &lt;td&gt;23.744963&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;4&lt;/th&gt;
      &lt;td&gt;ACY&lt;/td&gt;
      &lt;td&gt;20.344538&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;5&lt;/th&gt;
      &lt;td&gt;PSE&lt;/td&gt;
      &lt;td&gt;19.000000&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;6&lt;/th&gt;
      &lt;td&gt;CPR&lt;/td&gt;
      &lt;td&gt;18.383459&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;7&lt;/th&gt;
      &lt;td&gt;CWA&lt;/td&gt;
      &lt;td&gt;18.002533&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;8&lt;/th&gt;
      &lt;td&gt;SCE&lt;/td&gt;
      &lt;td&gt;16.741325&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;th&gt;9&lt;/th&gt;
      &lt;td&gt;ATW&lt;/td&gt;
      &lt;td&gt;16.138655&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;h2 id="another-example-bucketing"&gt;Another example: bucketing&lt;/h2&gt;
&lt;p&gt;&lt;a href="bucketing"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Why stop there? After seeing lots of use cases in the wild where code reuse and composability would make analysts a great deal more productive, we'll be working to build more and more high level analytics tools into Ibis.&lt;/p&gt;
&lt;p&gt;As another example which I present as mostly code-only, consider the task of creating a discrete binning of arrival delays and computing a number of metrics. I'll let the code speak for itself:&lt;/p&gt;
&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[11]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[11]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
       count    nulls   min   max        sum      mean  approx_nunique
0  120947440  2587529 -1437  2598  852674931  7.049963            3801
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[12]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;delay_buckets&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="nb"&gt;bin&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;
       &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
       &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay_buckets&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
               &lt;span class="n"&gt;include_over&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
               &lt;span class="n"&gt;include_under&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;delay_bin&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;delay_summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;airlines&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;bin&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arrdelay&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;delay_summary&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[12]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   delay_bin     count    nulls   min   max        sum        mean  \
0          3   4039142        0    60   120  333493735   82.565489   
1          4   1594279        0   121  2598  290961107  182.503255   
2          1  44759224        0     1    29  451748992   10.092869   
3          0  63108269  2587529 -1437     0 -531724696   -8.425595   
4          2   7446526        0    30    59  308195793   41.387862   

   approx_nunique  
0              62  
1            2490  
2              29  
3             471  
4              30  
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="cell border-box-sizing code_cell vbox"&gt;
&lt;div class="input hbox"&gt;
&lt;div class="prompt input_prompt"&gt;
In&amp;nbsp;[13]:
&lt;/div&gt;
&lt;div class="input_area box-flex1"&gt;
&lt;div class="highlight-ipynb"&gt;&lt;pre class="ipynb"&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;bin_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay_summary&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;delay_bin&lt;/span&gt;
            &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;On time&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Less then 30 min&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="s1"&gt;&amp;#39;30 min to 1 hour&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
                    &lt;span class="s1"&gt;&amp;#39;1 to 2 hours&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="s1"&gt;&amp;#39;More than 2 hours&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;delay_summary&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bin_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;bin_name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;delay_bin&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;div class="vbox output_wrapper"&gt;
&lt;div class="output vbox"&gt;


&lt;div class="hbox output_area"&gt;&lt;div class="prompt output_prompt"&gt;
    Out[13]:&lt;/div&gt;
&lt;div class="box-flex1 output_subarea output_pyout"&gt;


&lt;pre&gt;
   delay_bin     count    nulls   min   max        sum        mean  \
0          0  63108269  2587529 -1437     0 -531724696   -8.425595   
1          1  44759224        0     1    29  451748992   10.092869   
2          2   7446526        0    30    59  308195793   41.387862   
3          3   4039142        0    60   120  333493735   82.565489   
4          4   1594279        0   121  2598  290961107  182.503255   

   approx_nunique           bin_name  
0             471            On time  
1              29   Less then 30 min  
2              30   30 min to 1 hour  
3              62       1 to 2 hours  
4            2490  More than 2 hours  
&lt;/pre&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;
&lt;div class="text_cell_render border-box-sizing rendered_html"&gt;
&lt;p&gt;In future posts I'll go into some more detail on category types in Ibis (like the result of &lt;code&gt;bucket&lt;/code&gt;) and some of the other tools used here.&lt;/p&gt;
&lt;/div&gt;&lt;/p&gt;</summary></entry><entry><title>Using other compute engines with Ibis</title><link href="http://www.ibis-project.org/other-compute-engines/" rel="alternate"></link><updated>2015-08-02T22:22:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-08-02:other-compute-engines/</id><summary type="html">

&lt;p&gt;Several people have asked me about using Ibis with execution engines other than
Impala. The purpose of this post is to explain how one can make Ibis work with
other systems and what that might mean for the actual users.&lt;/p&gt;


&lt;h2&gt;Context&lt;/h2&gt;
&lt;p&gt;A primary goal of Ibis is to enable users to productively solve data analysis
problems at any scale using 100% Python code, while leveraging the existing
ecosystem of 3rd party libraries and high performance computing tools as much
as possible. Personally, I'd like to see folks using tools like Cython to
accelerate mission-critical analytics on a petabyte of data, and we simply are
not there yet.&lt;/p&gt;
&lt;p&gt;"100% Python" is not a gimmick; it means that Ibis, among other things, will
also need to enable full utilization of the data analysis capabilities provided
by SQL without writing any actual SQL code. The "writing SQL [in Python]
without writing SQL" is such an important, and frankly nuanced, topic that I'm
going to write a dedicated follow up blog post about it.&lt;/p&gt;
&lt;p&gt;Starting in late 2014, I began developing a joint roadmap with the Impala team
at Cloudera to solve the painful performance and usability problems that Python
programmers experience at large scale (multi-terabyte to petabyte
workloads). Since I've been focused on support and integration with Impala
(which I understand not everyone has available yet), people have asked me about
using Ibis with other systems like:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Other SQL-on-Hadoop compute engines: Hive, Presto, and others&lt;/li&gt;
&lt;li&gt;Other "full-stack" (storage + compute) SQL engines: PostgreSQL, MySQL,
  Vertica, etc.&lt;/li&gt;
&lt;li&gt;Generalized compute engines: Spark&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Short answer: yes, if someone writes an Ibis expression translator for these
platform targets, they can be used with them with no problems. Where things get
dicey is how to make Python &lt;em&gt;user-defined functions&lt;/em&gt; (which can't be compiled
to SQL or some other form) run fast (or run at all); something that we are
doing internal work in Impala to enable.&lt;/p&gt;
&lt;p&gt;I'm really excited in particular to see more SQL-based backends built for Ibis
so that data analysts can just use Python instead of a mix of Python and SQL
(the unfortunate status quo). It's not as much work as you might think.&lt;/p&gt;
&lt;h2&gt;Building new SQL backends for Ibis&lt;/h2&gt;
&lt;p&gt;Ibis contains a fully-decoupled data expression API that implements a &lt;em&gt;superset
of SQL semantics&lt;/em&gt;. My view is that if you can write it in SQL, you should be
able to write it in a composable, reusable, and testable Pythonic (or
pandas-ic?) way with Ibis.&lt;/p&gt;
&lt;p&gt;When you build and execute an Ibis expression using Impala SQL, here's what
happens:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;The expression is analyzed and validated (types and relational semantics)
   while you're building it. So when you want to execute it you can be sure that
   it is valid. All of this code lives in &lt;code&gt;ibis/expr&lt;/code&gt; in the codebase. I'll
   definitely write some follow up posts and documentation going into detail on
   the expression API design.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The expression is "compiled" to identify each component corresponding to the
   parts of a SQL statement. See &lt;code&gt;ibis/sql/compiler.py&lt;/code&gt; (NB. this module may
   move in the future).&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;In this step, further analysis takes place to expand higher-level Ibis
  constructs into concrete low-level SQL primitives.&lt;/li&gt;
&lt;li&gt;Note this even handles all the SQL "weird stuff" like correlated
  subqueries; I'll write more technical detail about this in the future.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The compiled SELECT data structure is translated into a valid SQL query. This
   falls into two buckets of functionality:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;ibis/sql/ddl.py&lt;/code&gt;: the primary SQL statement builder classes.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ibis/sql/exprs.py&lt;/code&gt;: translates Ibis operations and arguments into
  concrete SQL function calls. This includes all built-in analytical
  functions functions, case statements, "synthetic" operations like
  bucketing, window functions, and so forth.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;To give you a concrete example, the Ibis code&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;37&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float_col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;38&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;double_pct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;double_col&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;double_col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;39&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
   &lt;span class="o"&gt;....&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;         &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bucket&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;string_col&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
   &lt;span class="o"&gt;....&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;         &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;In&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;expr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;Out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
   &lt;span class="n"&gt;bucket&lt;/span&gt; &lt;span class="n"&gt;string_col&lt;/span&gt;  &lt;span class="n"&gt;count&lt;/span&gt;  &lt;span class="n"&gt;nulls&lt;/span&gt;       &lt;span class="nb"&gt;min&lt;/span&gt;       &lt;span class="nb"&gt;max&lt;/span&gt;       &lt;span class="nb"&gt;sum&lt;/span&gt;      &lt;span class="n"&gt;mean&lt;/span&gt;  &lt;span class="n"&gt;approx_nunique&lt;/span&gt;
&lt;span class="mi"&gt;0&lt;/span&gt;       &lt;span class="mi"&gt;0&lt;/span&gt;          &lt;span class="mi"&gt;1&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000030&lt;/span&gt;  &lt;span class="mf"&gt;0.000030&lt;/span&gt;  &lt;span class="mf"&gt;0.022222&lt;/span&gt;  &lt;span class="mf"&gt;0.000030&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;1&lt;/span&gt;       &lt;span class="mi"&gt;0&lt;/span&gt;          &lt;span class="mi"&gt;4&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000122&lt;/span&gt;  &lt;span class="mf"&gt;0.000122&lt;/span&gt;  &lt;span class="mf"&gt;0.088889&lt;/span&gt;  &lt;span class="mf"&gt;0.000122&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;2&lt;/span&gt;       &lt;span class="mi"&gt;1&lt;/span&gt;          &lt;span class="mi"&gt;6&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000183&lt;/span&gt;  &lt;span class="mf"&gt;0.000183&lt;/span&gt;  &lt;span class="mf"&gt;0.133333&lt;/span&gt;  &lt;span class="mf"&gt;0.000183&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;3&lt;/span&gt;       &lt;span class="mi"&gt;1&lt;/span&gt;          &lt;span class="mi"&gt;9&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000274&lt;/span&gt;  &lt;span class="mf"&gt;0.000274&lt;/span&gt;  &lt;span class="mf"&gt;0.200000&lt;/span&gt;  &lt;span class="mf"&gt;0.000274&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;4&lt;/span&gt;       &lt;span class="mi"&gt;0&lt;/span&gt;          &lt;span class="mi"&gt;3&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000091&lt;/span&gt;  &lt;span class="mf"&gt;0.000091&lt;/span&gt;  &lt;span class="mf"&gt;0.066667&lt;/span&gt;  &lt;span class="mf"&gt;0.000091&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;5&lt;/span&gt;       &lt;span class="mi"&gt;0&lt;/span&gt;          &lt;span class="mi"&gt;0&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000000&lt;/span&gt;  &lt;span class="mf"&gt;0.000000&lt;/span&gt;  &lt;span class="mf"&gt;0.000000&lt;/span&gt;  &lt;span class="mf"&gt;0.000000&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;6&lt;/span&gt;       &lt;span class="mi"&gt;1&lt;/span&gt;          &lt;span class="mi"&gt;5&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000152&lt;/span&gt;  &lt;span class="mf"&gt;0.000152&lt;/span&gt;  &lt;span class="mf"&gt;0.111111&lt;/span&gt;  &lt;span class="mf"&gt;0.000152&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;7&lt;/span&gt;       &lt;span class="mi"&gt;1&lt;/span&gt;          &lt;span class="mi"&gt;7&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000213&lt;/span&gt;  &lt;span class="mf"&gt;0.000213&lt;/span&gt;  &lt;span class="mf"&gt;0.155556&lt;/span&gt;  &lt;span class="mf"&gt;0.000213&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;8&lt;/span&gt;       &lt;span class="mi"&gt;1&lt;/span&gt;          &lt;span class="mi"&gt;8&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000244&lt;/span&gt;  &lt;span class="mf"&gt;0.000244&lt;/span&gt;  &lt;span class="mf"&gt;0.177778&lt;/span&gt;  &lt;span class="mf"&gt;0.000244&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="mi"&gt;9&lt;/span&gt;       &lt;span class="mi"&gt;0&lt;/span&gt;          &lt;span class="mi"&gt;2&lt;/span&gt;    &lt;span class="mi"&gt;730&lt;/span&gt;      &lt;span class="mi"&gt;0&lt;/span&gt;  &lt;span class="mf"&gt;0.000061&lt;/span&gt;  &lt;span class="mf"&gt;0.000061&lt;/span&gt;  &lt;span class="mf"&gt;0.044444&lt;/span&gt;  &lt;span class="mf"&gt;0.000061&lt;/span&gt;               &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;internally runs the SQL query&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;string_col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;nulls&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="k"&gt;min&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="k"&gt;max&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;avg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ndv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;approx_nunique&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;CASE&lt;/span&gt;
      &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;float_col&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;float_col&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
      &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;float_col&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;float_col&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
      &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
    &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;bucket&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;double_col&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="k"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;double_col&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;OVER&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;double_pct&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;ibis_testing&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;&lt;span class="n"&gt;functional_alltypes&lt;/span&gt;&lt;span class="o"&gt;`&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Ibis's SQL translation toolchain is extensively tested in &lt;code&gt;ibis/sql/tests&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Adapting Ibis to work with other systems that use the SQL language primarily
involves addressing engine-specific differences in Step 3. There may be some
nuances in Step 2; as a concrete example, not all databases have explicit
semi-joins, which Ibis uses to adapt Top-N filter operations.&lt;/p&gt;
&lt;p&gt;If you want to enable Ibis to use a SQL engine that you have available, like
Vertica or PostgreSQL, I am happy to guide you in the efforts to begin building
a test suite and refactoring Ibis's SQL translation subsystem to target
multiple SQL dialects. The hard part (and why I won't promise to build the
integrations myself) is thorough integration testing; think of it as "code
coverage" for SQL. It's not enough to generate the queries; you have to make
sure they work.&lt;/p&gt;
&lt;h2&gt;Executing Ibis expressions with pandas?&lt;/h2&gt;
&lt;p&gt;This is hypothetically possible, but there are some feature gaps in pandas, and
pandas's loose type system (and known issues with &lt;code&gt;NULL&lt;/code&gt; / &lt;code&gt;NA&lt;/code&gt; values) would
be a barrier. If someone wants to discuss this with me in more detail please
get in touch.&lt;/p&gt;
&lt;p&gt;I expect that an in-memory backend for Ibis will exist at some point in the
future, but it's unlikely to use pandas. Interoperability with pandas is a
separate issue, and using Ibis alongside pandas is a primary use case.&lt;/p&gt;
&lt;h2&gt;Why Ibis doesn't use SQLAlchemy (yet)&lt;/h2&gt;
&lt;p&gt;Several people have asked me, "Wes, why aren't you using
&lt;a href="http://www.sqlalchemy.org/"&gt;SQLAlchemy&lt;/a&gt; to generate SQL code in Ibis? It does
away with all the database-specific differences!"  This is only half true. Here
are the main reasons:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Impala will soon have complex types and its own SQL syntax for expressing
  queries involving arrays, structs, and maps as primitive value types. Engines
  like Presto also have this functionality and their own syntax for writing
  queries on such data. SQLAlchemy support for complex types is unlikely to
  drop in time (i.e. the next 6 months or sooner).&lt;/li&gt;
&lt;li&gt;We have to write interface code to wrap all the SQL built-in functions, and
  many of these are database specific. SQLAlchemy provides an extension API for
  wrapping functions, but it was easier, at least initially, to handle all the
  function wrapping within Ibis.&lt;/li&gt;
&lt;li&gt;Impala, like Hive, has various big data-specific DDL commands and modifiers;
  we are generating more than only &lt;code&gt;SELECT&lt;/code&gt;, &lt;code&gt;INSERT&lt;/code&gt;, and &lt;code&gt;CREATE TABLE&lt;/code&gt;
  statements.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That being said, I would like to see a SQLAlchemy version of Ibis's SQL
translation toolchain, in particular for targeting traditional RDBMSs like
PostgreSQL or MySQL.&lt;/p&gt;
&lt;p&gt;SQLAlchemy handles a number of things, like input sanitization, that Ibis does
not yet, so there are likely some code reuse opportunities there.&lt;/p&gt;
&lt;h2&gt;Why Impala is so key to the Ibis vision&lt;/h2&gt;
&lt;p&gt;Among the production-grade open source big data systems one may consider,
Impala is the only one that does not run on the JVM. So, SQL performance /
interactivity considerations aside, we (myself and the Impala team) are working
on shared memory / binary interoperability with Python that eliminates the data
serialization and memory use overhead that has plagued Python extension APIs
(for user-defined functions) on top of JVM-based runtimes. Most importantly, it
will allow users to use scientific Python tools (like Cython) to write high
performance code that operates on data flowing around the Impala C++/LLVM
runtime. If you have to pay a high transfer cost to move data to and from
Python, the benefits of such tools can be largely negated in many use cases.&lt;/p&gt;
&lt;h2&gt;Will Ibis be useful without Impala&lt;/h2&gt;
&lt;p&gt;Oh, yes, incredibly useful. The value of writing Python code (high level)
instead of SQL code (low level) is extremely high. More on this in a future
post.&lt;/p&gt;
&lt;h2&gt;About Blaze (Python project)&lt;/h2&gt;
&lt;p&gt;&lt;a href="http://blaze.pydata.org"&gt;Blaze&lt;/a&gt; (from my friends at &lt;a href="http://continuum.io"&gt;Continuum Analytics&lt;/a&gt;) is a Python project with &lt;em&gt;some&lt;/em&gt; overlapping
goals with Ibis. Both use the same general technical approach of building a
decoupled expression API with a separate compiler-executor. Overall I would
describe them as providing two distinct domain specific languages for
structured data having different priorities (based on what I see on GitHub).&lt;/p&gt;
&lt;p&gt;While it's too nuanced for this blog post, I will state my views on how Blaze
is similar to / different from Ibis as follows:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Blaze has varying levels of support for &lt;em&gt;lots of backends&lt;/em&gt;. The price of this
  is that any given backend is likely missing substantial functionality in
  Blaze's expression API (I know this to be true from reading the test
  suite). We started Ibis with comprehensive coverage of &lt;em&gt;one backend&lt;/em&gt; (Impala,
  for which we are developing an efficient Python extension API) as a blueprint
  for additional backends to be contributed by the community.&lt;/li&gt;
&lt;li&gt;I don't believe it was ever a goal of Blaze to replace SQL (the language) in
  user workflows.&lt;/li&gt;
&lt;li&gt;Blaze's expression API is designed fairly differently. I may do an in-depth
  comparison of how the differences play out in real code (sooner if a lot of
  people are interested).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Complete coverage of the functionality provided by &lt;em&gt;one&lt;/em&gt; SQL engine is
truthfully a fairly difficult problem. There are canonical database benchmark
suites like TPC-H and TPC-DS that you can use to put a lot of stress on any
given DSL that can emit SQL queries.&lt;/p&gt;
&lt;p&gt;If you're trying to decide which project to use, look at the projects (the test
suites, in particular) and make your own decisions.&lt;/p&gt;
&lt;h2&gt;Summary&lt;/h2&gt;
&lt;p&gt;With the current completeness of the Impala SQL backend in Ibis, the work of
integrating with other data engines that speak SQL is relatively
straightforward and incremental. I will happily welcome well-tested
contributions to the project.&lt;/p&gt;
&lt;p&gt;In an upcoming blog post, I will write in some more detail about how Ibis will
help bring about a post-SQL world and why that will be a productivity boon for
data analysts everywhere.&lt;/p&gt;</summary></entry><entry><title>Hello World</title><link href="http://www.ibis-project.org/hello-world/" rel="alternate"></link><updated>2015-07-23T19:41:00-07:00</updated><author><name>Wes McKinney</name></author><id>tag:www.ibis-project.org,2015-07-23:hello-world/</id><summary type="html">

&lt;p&gt;Welcome to the Ibis project blog. As the project develops we'll post here
periodically with:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Applications and use cases&lt;/li&gt;
&lt;li&gt;Development updates and release notes&lt;/li&gt;
&lt;li&gt;Details on project internals and design&lt;/li&gt;
&lt;li&gt;Updates from Impala development and the broader community&lt;/li&gt;
&lt;li&gt;Thought pieces on the Big Data and Python ecosystems&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I'm looking forward to the journey; it's going to be really exciting.&lt;/p&gt;
</summary></entry></feed>