FireDucks – Posts

Posts: How to take traces in FireDucks

Fri, 20 Dec 2024 00:00:00 +0000

FireDucks has a trace function that records how long each process such as read_csv, groupby, sort, etc. takes. This article introduces how to use the trace function.

How to output and display trace files

To use the trace function, you do not need to modify the program. Simply set the environment variables as shown below and execute the program to use the trace function.

$ FIREDUCKS_FLAGS="--trace=3" python -mfireducks.pandas your_program.py

After setting the environment variables and executing the program, a file named trace.json is created in the directory where the program was executed. This file is the trace file.

To view a trace file, use either Microsoft Edge or Google Chrome, a web browser with trace viewer functionality. You can start the trace viewer by typing edge://tracing for Microsoft Edge or chrome://tracing for Google Chrome in the address bar.

The following image shows the Trace Viewer running in Microsoft Edge.

Click the Load button to open the trace file. The execution trace of the program will be displayed graphically. The following image shows the execution trace of one query of the polars-tpch benchmark introduced in this article.

The top shows the time of the whole program (or, more correctly, the time between import fireducks.pandas and the end of the program). Below that, fireducks.core.evaluate is divided into two major blocks. The polars-tpch benchmark run explicitly separates the reading of the parquet file and the execution of the query. Therefore, the evaluation is split into two.

In the first half of the evaluation, you can see that only the fireducks.read_parquet_with_metadata parquet reading process accounts for the execution time. You can also zoom in with the mouse to get a more detailed breakdown of the execution time for the second half of the query, as shown below.

How to change the trace file name

The default trace file name is trace.json, but you can set an arbitrary file name as follows: --trace-file=foo.json.

$ FIREDUCKS_FLAGS="--trace=3 --trace-file=foo.json" python -mfireducks.pandas your_program.py

How to output trace summary to standard error

If you only want to see a breakdown of the time spent on each process, you can also display summary information in standard error.

The summary is displayed using the same options described above. Use --trace-file=- instead of the file name.

$ FIREDUCKS_FLAGS="--trace=3 --trace-file=-" python -mfireducks.pandas your_program.py

This is an example of the execution of the polars-tpch benchmark query used in the previous example. Although details such as the order of execution of each process are not available, a summary of the execution time can be viewed.

elapsed 6.071 sec
kernels 5.963 sec 98.22% 101
fallbacks 0.000 sec 0.00% 0
 duration sec ratio count
== kernel ==
fireducks.read_parquet_with_metadata 5.453 89.83% 1
fireducks.filter 0.293 4.83% 1
fireducks.groupby_agg 0.089 1.46% 1
fireducks.le.vector.scalar 0.051 0.83% 1
fireducks.mul.vector.vector 0.042 0.69% 2
fireducks.rsub.vector.scalar 0.023 0.38% 1
fireducks.radd.vector.scalar 0.009 0.15% 1
fireducks.sort_values 0.002 0.03% 1
fireducks.read_parquet_metadata 0.001 0.02% 1
fireducks.project 0.000 0.00% 8
== fallback ==
== other ==
top 6.071 100.00% 1
create_mlir_func 0.001 0.02% 3
import pandas 0.000 0.00% 2
fire.get_string 0.000 0.00% 22

Conclusion

This article has introduced how to use the trace function in FireDucks.

When using FireDucks, there may be times when you notice a slowdown. In that case, you may be able to find the process that caused the slowdown by tracing with the help of this article.

We hope that you will make full use of FireDucks by using the trace function.

Posts: Ensuring compatibility with pandas in the GPU version of FireDucks

Thu, 19 Dec 2024 00:00:00 +0000

We are currently developing a GPU version of FireDucks.

FireDucks is built with an architecture that translates programs into an intermediate representation at runtime, optimizes them in this intermediate representation, and then compiles and executes the intermediate representation for the backend. The currently released CPU version of FireDucks has a backend for CPUs. In the development of the GPU version, the backend is changed to a GPU. This allows us to use the translation to and optimization of the intermediate representation developed for the CPU version as is.

In developing the GPU backend, we are making use of NVIDIA’s cuDF library. Our intermediate representation is roughly compatible with the pandas API, and cuDF provides similar API to pandas, so at first glance the backend development seems straightforward. However, the functions provided by cuDF are slightly different from those provided by pandas, so some ingenuity is required to maintain compatibility with pandas.

In this blog post, I would like to briefly introduce some of the issues that need to be addressed in order to maintain compatibility with pandas.

Different result types

When working with dates in pandas, if you convert them to the datetime64 type, you can use the dt accessor to extract the year, month, and day.

df = pd.DataFrame({"a": ["2017-11-01 12:24:00"]})
dfa = pd.to_datetime(df["a"])
print(dfa.dt.year)

will return 2017. By the way, what is the type of this value? If you do print(dfa.dt.year.dtype), pandas returns int32 and cuDf returns int16. This difference seems to be intentional, in order to reduce the amount of memory used by the GPU.

The year is unlikely to exceed 16 bits, and there doesn’t seem to be a major problem, but if you use it for calculations, there is a possibility that the result will change due to overflow. For example, if you try to calculate the number of hours from year 0, and use dfa.dt.year * 365 * 24, you will get a different result because of the overflow in int16.

This is not limited to this example, but in cuDF, the type of the result is often slightly different. In FireDucks, we convert the result to the same type to maintain compatibility with pandas.

Missing values are handled differently in calculations

In pandas, missing values (what we call NULL in RDB) are basically expressed as NaN. On the other hand, in cuDF, missing values are treated as a special value (NA). Therefore, the calculation results may be different.

For example,

df = pd.Series([1.0, 3.0, np.nan])
mask = df < 2.0
print (mask)

will print

0 True
1 False
2 False

in the case of pandas. False is entered in the missing value position. This is because it is compared with np.nan.

On the other hand, in the case of cuDF,

0 True
1 False
2 <NA>

is printed. The result of the operation is also a missing value (NA). In RDB, operations with NULL always result return NULL, so the results are consistent with RDB, but not with pandas. I think this is also an intentional feature of cuDF.

In pandas, a special value (pd.NA) can be used experimentally as a missing value. When this is used, the results are the same as in cuDF and RDB, but otherwise the results are different. In FireDucks, the results are adjusted so that the handling of missing values is the same as pandas.

The results of merge are different

Pandas’ merge has a complex specification, and cuDF does not necessarily follow that specification. This may change in the future, but for now there are differences, such as the following.

In pandas merge, you can use left_on and right_on to specify the columns to be used for merging. Normally, you specify the column names here, but if the index has a name, you can also specify the index name. You can also specify the index by specifying left_index=True or right_index=True. Let’s try merging using these functions.

First, create a DataFrame for merging. The left dataframe is created as:

idx1 = pd.Index([1,2,3,4],name="p")
df1 = pd.DataFrame([[1,2],[3,4],[5,6],[7,8]], columns=["a","b"], index=idx1)

The result is:

The right dataframe is created as:

idx2 = pd.Index([1,2,5,6],name="q")
df2 = pd.DataFrame([[3,4],[5,6],[1,2],[3,4]], columns=["c","d"], index=idx2)

The result is:

Now let’s try merging these. On the left, we specify the index column by name with left_on=[“p”], and on the right, we specify that we want to use the index column with right_index=True. We specify “outer” for how to perform an outer join, as in an RDB.

df1.merge(df2, left_on=["p"], right_index=True, how="outer")

In the case of pandas, the result is:

 p a b c d
1.0 1 1.0 2.0 3.0 4.0
2.0 2 3.0 4.0 5.0 6.0
3.0 3 5.0 6.0 NaN NaN
4.0 4 7.0 8.0 NaN NaN
NaN 5 NaN NaN 1.0 2.0
NaN 6 NaN NaN 3.0 4.0

In the case of cuDF,

 a b c d
p
1 1 2 3 4
2 3 4 5 6
3 5 6 <NA> <NA>
4 7 8 <NA> <NA>
<NA> <NA> <NA> 1 2
<NA> <NA> <NA> 3 4

The type of the column with missing values is different, because pandas uses NaN for missing values, which causes the type of the column float64 type. The big difference is that pandas has a column called p, but cuDF does not. Also, the name of the resulting index is different.

The column p is originally the index column of the left DataFrame, and it seems like a strange specification that this is created as a column of the resulting DataFrame (columns like p are only created in the case of the special parameter combination described above). In RDB joins, such columns are not created. On the other hand, the results are clearly different, so there is a possibility that this could cause a compatibility problem for user programs.

In FireDucks, we have adjusted the code so that the results are as close as possible to those of pandas even in such cases.

As described above, there are cases where the results differ between pandas and cuDF. The GPU version of FireDucks has been implemented to absorb such differences and produce results as close as possible to those of pandas. The GPU version of FireDucks is still under development and is not yet ready for use by users, but we hope that you will try it out when it is completed.

Posts: Cache or Eliminate? How FireDucks increase opportunity of optimization

Tue, 17 Dec 2024 00:00:00 +0000

As described here, FireDucks uses lazy execution model with define-by-run IR generation. Since FireDucks uses MLIR compiler framework to optimize and execute IR, first step of the execution is creating MLIR function which holds operations to be evaluated. This article describes how important this function creation step is for optimization, thus performance.

In the simple example below, execution of IR is kicked by the print statement which calls df2.__repr__().

df0 = pd.read_parquet("date.parquet", [])
df1 = df0.sort_value("a")
df2 = df1[["b", "c"]]
print(df2)
 :

When the execution is started, FireDucks collects operations which are required to compute df2 and put those into a MLIR function. Simplest MLIR function would be as below.

// MLIR function (simplified)
func main() {
 %t0 = read_parquet("data.parquet", [])
 %t1 = sort_values(%t0, ["a"], [True])
 %t2 = project(%t1, ["b", "c"])
 return %t0, %t1, %t2
}

In this function, the return op at the end of the function returns three values, %t0, %t1 and %t2. Returned values will be bound to the variables in python, i.e. df0, df1 and df2, after the execution. If such variables will be used after print statement, those results will be used. One can say that the result of the operations are cached to avoid re-execution.

This function is however very strict for IR optimization. Because all output values of ops, %t0, %t1 and %t2, are returned, an optimizer has to preserve these values. On the other hand, for example, if return op returns only %t2, an optimizer can change IR as below. In this IR, only three column "a", "b", "c" will be read from a file by read_parquet op to minimize reading time and memory footprint because the result of it, %t0, is only used to compute %t2. As this simple example shows, because returning values from a function restricts optimization opportunity, returned values have to be carefully chosen.

// MLIR function (simplified)
func main() {
 %t0 = read_parquet("data.parquet", ["a", "b", "c"]) // only three columns are read from a parquet file.
 %t1 = sort_values(%t0, ["a"], [True])
 %t2 = project(%t1, ["b", "c"])
 return %t2
}

Liveness analysis

To this end, FireDucks uses liveness analysis of python variables when creating a function. For example, if print statement is the last line of a python script, the liveness analysis can say that df0 and df1 are dead at the print statement because no user of those variables is in the script after the statement. With this analysis, FireDucks can eliminate %t0 and %t1 from the return op. As an another example, if df1 is used after print statement, the liveness analysis says that it is live, and %t1 is not eliminated from return op.

In lazy execution system, re-execution sometimes drastically decreases performance. By using liveness analysis, FireDucks detects whether variables are used after execution or not in order to increase optimization opportunity with avoiding re-execution. AFAIK, this is the reason why FireDucks is very faster than polars in this article. You can reproduce it on colab.

How does it work in a notebook?

You may wonder how liveness analysis works when you are writing in a notebook or ipython. In such environment, because all python variables might be used in future cells, liveness analysis has to say that all variables are live. This results in limited optimization opportunity.

One workaround is that using chained style:

pd.read_parquet("date.parquet", []).sort_values("a")[["b", "c"]]

In this style, because results of intermediate operations, read_parquet and sort_values are not bound to any python variables, there is no chance to be used in future cells. In this case, FireDucks’s liveness analysis can say that those are dead when evaluating the result of last operation.

Posts: How to run polars-tpch benchmark with FireDucks

Fri, 06 Dec 2024 00:00:00 +0000

Recently we have updated the result of polars-tpch benchmark on 4th generation Xeon processor. The latest result can be found here, and also below in this artice, explaining how to reproduce the same.

For reproducibility, we have used AWS EC2 for this time evaluation. We have used m7i.8xlarge instance type with ubuntu 24.04 image and 128GB EBS SSD. This instance includes:

4th generation Xeon processor: Intel(R) Xeon(R) Platinum 8488C (32cores)
128GB memory

Benchmark Result

The graph shown below compares performance of four dataframe libraries, pandas, polars, modin and fireducks as a speedup from pandas. For an average of 22 queries:

fireducks is 125x faster than pandas, whereas
polars is 57x faster than pandas,
modin is 1.0x faster than pandas

Note that our setting of benchmark is as follows:

Scale factor is 10 (about 10GB dataset) SCALE_FACTOR=10.0
Timings without IO RUN_IO_TYPE=skip

How to run the benchmark with FireDucks

After launching instance and logging into it, you need to install python:

$ sudo apt update
$ sudo apt install python3.10-venv make gcc

Then clone the benchmark code from our repository which includes queries for FireDucks:

$ git clone https://github.com/fireducks-dev/polars-tpch
$ cd polars-tpch

Then run the script which first creates dataset with SCALE_FACTOR=10.0 and runs all 22 queries once with FireDucks as per the above settings:

$ ./run-fireducks.sh

Now you can see timings for all queries in output/run/timings.csv.

How to produce timings for the graph

The graph shown above was created with minimum execution time among three runs for each query.

To do this:

$ SCALE_FACTOR=10.0 make tables # create dataset by polars
$ .venv/bin/pip install -U pandas polars modin # install latest libraries
$ ./run-fppm3.sh # run four libraries three times

How Queries for FireDucks Are Implemented

When we have started to use TPC-H benchmark for evaluation of FireDucks, we have implemented pandas version of queries so that those produce the correct result. Later, when we came to know about polars-tpch benchmark, we have updated our queries as like polars as possible, for example same number of operations, same order of operations, etc.

Since both libraries have different APIs, implementations are not 100% same, but to my surprise both implementations look very similar, thanks to flexible pandas API. Here is query 2 as an example. Can you figure out which one is polars and which one is pandas?

In this post, we have described how to reproduce our results for TPC-H Benchmark. Let’s try and take a look at both queries. If you find something to be improved for reasonable comparison among libraries, let us know.

Posts: What to do when FireDucks is slow

Mon, 11 Nov 2024 00:00:00 +0000

Thank you for your interest in FireDucks. This article describes possible causes and remedies for slow programs using FireDucks.

When a pandas program with FireDucks applied is slow, the reason may be the followings.

Using ‘apply’ or ’loop’.
Using pandas API not implemented in FireDucks.

In the case of 1, if you change the pandas program, the program may become faster. For example,

sum_val = 0
for i in range(len(df)):
 if df["A"][i] > 2:
 sum_val += df["B"][i]

A program using ’loop’ like the one above can be faster by writing it like the one below.

sum_val = df[df["A"] > 2]["B"].sum()

If you have difficulty in confirming by yourself, or if the source code is too complicated to find suitable modification methods, please feel free to ask for help from the FireDucks community via the slack listed below.

In the case of 2, we cannot immediately speed up the program, but by reporting unimplemented pandas API in FireDucks, we will implement them and speed up the program in the future.

Please set environment variables to determine if FireDucks implements the functionality you need for your pandas program.

FIREDUCKS_FLAGS="-Wfallback"

After setting the environment variables, please run the program. If you see the word “Fallback” in the pandas function, it would be helpful if you could report it.

If you would like to report a problem, please contact us by any of the following methods.

🦆github : https://github.com/fireducks-dev/fireducks/issues/new
📧mail : contact@fireducks.jp.nec.com
🤝slack : https://join.slack.com/t/fireducks/shared_invite/zt-34qpdgr6q-_iWdIoZW4l_hGhljKS0pyg

Please feel free to contact us even if you have difficulty in researching Fallback.

This concludes this article. Thank you for reading.

Posts: Workshop at Bangalore, India

Thu, 19 Sep 2024 00:00:00 +0900

We had a workshop on FireDucks with faculties from universities around Bangalore. Thank you for joining and discussion.

Posts: Have you ever thought of speeding up your data analysis in pandas with a compiler?

Mon, 01 Jul 2024 00:00:00 +0000

In general, a Data Scientist spends significant efforts in transforming the raw data into a more digestible format before training an AI model or creating visualizations. Traditional tools such as pandas have long been the linchpin in this process, offering powerful capabilities but not without limitations.

With the pitfall of its single-core implementation and inefficient data structures, often we face performance issues when dealing with pandas for relatively larger data, but its performance is also highly impacted by the choice of APIs, their parameters and execution orders. Sometime an efficient writing of a pandas application can itself make it 10-20x faster.

In this article, I will discuss a few commonly used examples picked up from some pandas applications.

:notebook: Example 01

df = pd.DataFrame()
s = pd.Series(["2020-10-10", "2021-11-20", "2022-08-03", "2023-07-04"])
df["year"] = pd.to_datetime(s).dt.year
df["month"] = pd.to_datetime(s).dt.month
df["day"] = pd.to_datetime(s).dt.day

We can clearly identify the major issue with the above code is using the expression, pd.to_datetime(s) three times. The purpose is to extract the year, month and day fields from the series data (s) of type “datetime64[ns]”. The method to_datetime() attempts to parse the input string column in order to convert it to a datetime column (along with inferring the format if not specified). Such parsing itself is very expensive when performed on a large data and if the same expression on the same input is written more than one time, its very obvious that the program will experience a significant performance issue.

As you might have clearly figured out, an optimized solution to this problem could be:

df = pd.DataFrame()
s = pd.Series(["2020-10-10", "2021-11-20", "2022-08-03", "2023-07-04"])
dt_s = pd.to_datetime(s)
df["year"] = dt_s.dt.year
df["month"] = dt_s.dt.month
df["day"] = dt_s.dt.day

This is a very basic example which can be manually optimized, if reviewed carefully after an application is developed. Let’s take another example which is very common when performing an extensive data analysis.

:notebook: Example 02

res = pd.DataFrame()

# To find industry-wise average salary of an Indian employee
res["industry_wise_avg_sal"] = (
 employee[employee["country"] == "India"]
 .groupby("industry")["salary"]
 .mean()
)

# To find industry-wise average salary of an Indian employee who are above 30
res["industry_wise_avg_sal_for_specific_age_group"] = (
 employee[(employee["country"] == "India") & (employee["age"] >= 30)]
 .groupby("industry")["salary"]
 .mean()
)

print(res)

Both of these queries have a common expression to be evaluated during the filter operation, i.e., checking whether the employee belongs to India (comparison on String column is itself very costly than comparison on numeric columns). Now, imagine if we are dealing with an extensively large employee database, then evaluating employee["country"] == "India" for two times can be quite an expensive operation that can easily be optimized with the same strategy as follows:

# To generate the required filtration masks in advance
cond1 = (employee["country"] == "India")
cond2 = (employee["age"] >= 30)

res = pd.DataFrame()
# To find industry-wise average salary of an Indian employee
res["industry_wise_avg_sal"] = (
 employee[cond1]
 .groupby("industry")["salary"]
 .mean()
)

# To find industry-wise average salary of an Indian employee who are above 30
res["industry_wise_avg_sal_for_specific_age_group"] = (
 employee[cond1 & cond2]
 .groupby("industry")["salary"]
 .mean()
)

print(res)

In compiler technology, such optimization is called as common sub-expression elimination (CSE) and can essentially be performed by a smart compiler in programming language like C/C++.

:notebook: Example 03

Let’s take another example of a compiler optimization technique:

def func(x: pd.DataFrame, y: pd.DataFrame):
 merged = x.merge(y, on="key")
 sorted = merged.sort_values(by="key") # is never used
 return merged.groupby("key").max()

The method is trying to merge two input tables, x and y followed by a groupby-aggregate operation. The “sort” operation is also a part of the method after merging the tables, but it is actually not required from the method context (since the sorted result is never used within the method). Sometime when we focus on very detail exploration of the input data, it is very common that such piece of unwanted code remains in our application resulting a significant performance cost.

Hence, an optimized solution to the above method could be:

def func(x: pd.DataFrame, y: pd.DataFrame):
 merged = x.merge(y, on="key")
 return merged.groupby("key").max()

In compiler technology, such optimization is called as dead code elimination and is very important from the overall performance point of view of any application.

An expert programmer might take care of such cases while developing his data analysis solutions, but as a data analyst our primary focus is to explore more about the data from different verticals and horizontals to find out meaningful insights while solving some business problems or creating important features for our training data. Hence we often miss to take care of such issues. Nowadays, there are many good linters that will point out such issues, so a manual optimization is definitely possible if you include the linter execution step as part of your TODO action when developing an application.

:point_right: New Information alert!

Let’s now talk about an ideal scenario!

How would it be if such optimization is automatically taken care by the python data manipulation library we use?

It can save us from taking care of such issues by ourselves and can significantly improve the performance of our application, right?

Well, the wait is over! Let me introduce a high-performance DataFrame library, named FireDucks with highly compatible pandas APIs, powered by MLIR（Multi-Level Intermediate Representation) framework to have such powerful compiler optimization abilities.

The library is carefully developed at NEC R&D laboratory over last 3 years and is freely available to be installed using “pip” under BSD licence since October, 2023.

:fire: :bird: What does it offer?

FireDucks is developed with the following points in mind:

Automatic query optimization ability with lazy-execution model.
- A data analyst can focus more on exploring data, while the inbuilt compiler of FireDucks can take care of the following:
  1. basic compiler optimizations, like common sub-expression elimination, dead code elimination etc.
  2. domain specific optimizations, like execution reordering, dropping unwanted columns in advance etc.
  3. pandas specific optimizations, like choice of right APIs when executing a query (by careful analysis of the application objective), choice of right parameters by avoiding unwanted operations (like sorting of result etc.)
A pandas user can flexibly adapt to this library without any new learning cost.
- FireDucks is highly compatible with pandas, so any pandas application can be optimized without any manual code changes. Doesn’t it sound quite promising?
The program can leverage all the available cores in the execution environment:
- The single-core execution issue in pandas is solved!

:computer: Demonstration

Here is a link for a test drive with a sample walkthrough notebook on Google Colab which shows how easy it is to start with FireDucks and its performance gain over pandas for a sample CSE use-case (Example 01 of this article).

With a low-spec execution environment having 2 cores, where pandas takes 1.3 seconds to execute a sample query, FireDucks (with multhreaded + compiler enabled) takes only 166 milli-seconds. Such speedup (~8x) without incurring any migration cost (no cost involves in rewritting the application from pandas to FireDucks) or any special hardware cost seems quite beneficial from the overall production cost point of view.

In case you are interested to see how it performs for popular benchmarks (like TPC-H, db-benchmark etc.) in comparison to other high-performance pandas alternatives, you may like to check this out.

:point_right: How does FireDucks work?

FireDucks comes with three powerful layers:

a python frontend highly compatible with pandas APIs
an in-built compiler to auto-detect and optimize the exsting performance issues in a user program
a multithreaded C++ backend with efficient parallel implementation of all the dataframe related operations like join, filter, groupby sort etc.

Unlike pandas that executes the library functions right after they are called, FireDucks creates some special instructions and keep them accumulated until there is some explicit request for the result (print result, do some reduction, to_csv etc.). Before the execution, the in-built compiler inspects all the accumulated instructions related to the result to be processed and performs some automatic optimization (as explained above) and then the optimized instructions are executed at the multithreaded kernel backend helping us to focus more on our analytical work and be more productive.

✍️ Conclusion

FireDucks shows promise by taking care of all the drawbacks associated with pandas and its compiler optimization technology makes it one of its kind. This article introduces FireDucks basic compilation optimization abilities as mentioned in (a). In upcoming articles, I will demonstrate other powerful optimization areas (b and c) automatically taken care of by FireDucks. If you like this article, please be with me to check them out as well. You may like to try FireDucks and share your feedback. I would love to answer to whatever queries you may have in mind with the best of my knowledge. The development team is very active and there is a new release almost every week with performance improvements, new features based on user requests, bug fixes etc. You may like to get in touch with them directly through the slack channel.

You may also like to checkout one of my previous articles on FireDucks salient features.

Posts: Introduction to FireDucks: Get performance beyond pandas with zero learning cost!

Wed, 08 May 2024 00:00:00 +0000

Posts: Backtesting Trading Strategies with Ease: An excursion with FireDucks

Fri, 22 Mar 2024 00:00:00 +0900

Posts: FireDucks: Diving into API Compatibility with Pandas

Thu, 21 Mar 2024 00:00:00 +0900

Posts: Choosing Your Data Champion: A Side-by-Side Look at FireDucks and Polars

Wed, 20 Mar 2024 00:00:00 +0900

Posts: Boosting Data Analysis with FireDucks

Tue, 05 Mar 2024 00:00:00 +0900

Posts: A hidden fact you must know when working with pandas

Sat, 16 Dec 2023 00:00:00 +0000

Posts: Tricks to improve computational performance of JOIN operation more than 10x

Mon, 11 Dec 2023 00:00:00 +0000

Posts: FireDucks - An economical and environment-friendly high-performance solution for your complex Data Analysis

Thu, 07 Dec 2023 00:00:00 +0000

Posts: One thing you might be doing wrong in pandas!!

Wed, 06 Dec 2023 00:00:00 +0000

Posts: Acceleration technology inside FireDucks

Tue, 05 Dec 2023 00:00:00 +0900

SWITCHING GROUPBY

In this article, we introduce the acceleration techniques of “groupby” used in FireDucks.

The groupby operation is one of the most fundamental and important operations in tabular data analysis. We can use the groupby operation to obtain important statistical properties such as the mean and variance of the data. We can also combine it with other operations to obtain new features.

FireDucks optimizes based on data characteristics for fast groupby operations. One such optimization is the automatic selection of groupby algorithms based on the number of groups. FireDucks’ groupby algorithm focuses on the number of groups of data and switches between an algorithm that is fast for data with a small number of groups (Algorithm A) and an algorithm that is fast for data with a large number of groups (Algorithm B). The number of groups indicates the type of data that make up the target column.

Consider the following tabular data.

	food	category
0	apple	fruit
1	carrot	vegetable
2	peach	fruit
3	onion	vegetable
4	grape	fruit

There are five elements that make up the “food” column: “apple”, “carrot”, “peach”, “onion”, and “grape”. Therefore, the number of groups in the “food” column is 5. Next, we see that there are two types of elements that make up the “category” column: fruit and vegetable. Therefore, the number of groups in the “category” column is 2.

Calculating the number of groups for large data sets is time-consuming. Therefore, FireDucks uses a statistical method to estimate the number of groups without obtaining an exact value for the number of groups¹. The estimation is performed as follows.

Extracts a random piece of data from a sequence of data of interest.
Record that data.
Again, take out one data at random from the column of data of interest.
That data is checked to see if it matches the recorded data and record.
Repeat operations 3 and 4 multiple times.

After repeating operations 3 and 4 multiple times, FireDucks estimates the number of groups of data in the group key of interest from the number of times the data was retrieved and the number of matches.

Evaluation

We measured the calculation speed using TPC-H, a benchmark that includes many processes related to data analysis.

“A" uses only Algorithm A for groupby operations, and “B” uses only Algorithm B for groupby operations. “auto" estimates the number of groups and automatically selects between Algorithm A and B for the groupby operation.

Processes for which there is little difference in execution time between Algorithm A and Algorithm B are omitted from the above graph. The graph shows that, with the exception of q10, the automatic selection algorithm is the faster of Algorithm A and Algorithm B.

The total shows the computation time for the entire TPC-H, including the processes excluded from the above graph for Algorithm A, Algorithm B, and the automatic selection algorithm. This indicates that the automatic selection algorithm is about 3 times faster than Algorithm A, and about 1.2 times faster than Algorithm B for TPC-H as a whole.

Reference

M. Bressan, E. Peserico, and L. Pretto. Simple set cardinality estimation through random sampling. CoRR, abs/1512.07901, 2015. ↩︎

Posts: Application example: Spicy MINT at Toyota Technical Development Corporation

Thu, 19 Oct 2023 00:00:00 +0000