1. Scalable, Reliable Storage with Erasure Coding

Distributed storage systems have traditionally adopted data replication to ensure high availability and durability. Data replication has the disadvantage of requiring additional storage capacity equal to the number of replicas. As the amount of stored data increases exponentially ^[17,18], erasure coding is being adopted to use storage capacity more efficiently. For erasure coding, Reed-Solomon (RS) ^[19] codes are widely used ^[20-23]. The (n, k) RS code, composed of two parameters, n and k, encodes one data block into n data blocks and k parity blocks, and even if up to k blocks are lost, the original data block can be restored. Therefore, data storage capacity can be drastically reduced compared to the data replication method.

2. Storage Capacity Provisioning Tool

In the past decade, several storage provisioning studies have been conducted that aimed to design cost-effective storage systems using SSD instead of HDDs ^[14,24-27]. Narayanan et al. ^[24] conducted a study on the significance of Solid-State Drives (SSDs) within enterprise-level computing systems by analyzing actual data traces from data centers. They delved into the cost-efficiency comparisons across diverse arrangements of SSDs and Hard Disk Drives (HDDs). Kim et al. ^[25] explored the challenge of determining the most efficient storage configuration for computing systems that utilize both SSDs and HDDs, ensuring they fulfill performance criteria. Furthermore, they examined the strategy behind the fluid allocation of workloads in hybrid storage setups. On the other hand, Byun et al. ^[14] delved into the complexities of evaluating the cost-efficiency of establishing a storage system that incorporates Computational Storage Devices (CSDs). Given the intricacies of CSDs, which are significantly more advanced than SSDs, pinpointing the design prerequisites, including the number of CSDs necessary for a computational node based on a CSD array, poses a considerable challenge. They scrutinized the computational prowess and input/output processing capabilities of prevalent CSDs, finding that (i) CSDs exhibit a wide range of performance capabilities, distinct from SSDs, and (ii) there are noticeable disparities in performance trends even among CSDs themselves.

3. CSDPLAN: Capacity Planning

This section provides a brief introduction to CSDPLAN. More details can be found in the CSDPLAN paper ^[14]

1) System Modeling: CSDPLAN ^[14] is a software tool that uses a mathematical analysis model to guide system and storage designers in constructing storage systems. CSDPLAN’s model requires the I/O and computation performance parameters of CSDs for a given application as input and produces the minimum number of CSDs that are cost-effective when compared to a conventional SSD-based storage system as output.

CSDPLAN considers the execution time of an analysis kernel (W ) on two systems as follows.

• SSD-array system (n) (T$_{\mathrm{SSD(}}$$_{n}$$_{\mathrm{)-array}}$): A system with a host with a block-based SSD array. It executes the analytic kernel using the n cores and memory of the host.

• CSD-array system (T$_{\mathrm{CSD-array}}$): A system with a CSD array on the host. It executes the analytic kernel using the memory and CPU of the CSDs.

The kernel’s execution time in an SSD-array system is modeled as data transfer time (T$_{\mathrm{SSD-tx}}$) and computation time (T$_{\mathrm{SSD(}}$$_{n}$$_{\mathrm{)-comp}}$$_{.}$). If M SSDs are equipped, and the data required to run the kernel is uniformly distributed, the data transfer time is reduced to $1/M$. However, the computation is not reduced because it is performed by one host CPU. However, the computation is not reduced because it is performed by a single host CPU. Therefore, the model for an SSD-array system is as follows.

In an SSD-array system with an array of M SSDs, the data transfer time can theoretically be reduced by $1/M$. However, if the bus connection becomes a bottleneck ^[28] the reduction is no longer possible. Therefore, CSDPLAN defines the number of SSDs (M) at which the bottleneck occurs in an SSD-array system as follows.

The model of CSD-array system is more efficient than the SSD-array system because each CSD is equipped with computing power, reducing both computation and data transfer time to $1/M$. Furthermore, without a shared bus between CSDs, there are no bottlenecks as the number of CSDs increases. To extend the model of CSD-array system, the following steps are required.

2) Solver: Finding the Break-Even Point: CSDPLAN deploys a solver to find the BEP for the number of CSDs in a CSD-array-based node. CSDPLAN’s solve takes the CSD’s performance characteristics as an input and returns an optimal number of CSDs as an output, which represents the following BEP where the CSD-array is more effective than the conventional compute node. Therefore, the following model represents the formula for T$_{\mathrm{SSD(}}$$_{n}$$_{\mathrm{)-array}}$ to exceed TCSD-array.

Solve the above inequality for M. We omit the details of the theorem.

Therefore, the BEP (M) function based on the internal I/O bandwidth and computational power of the CSD is as follows.

1. Overview of CSDPLAN-EC

Fig. 1 depicts an overview of CSDPLAN-EC. Like CSDPLAN, CSDPLAN-EC is a software tool that provides storage architects with a break-even point (BEP). CSDPLAN-EC is specifically extended to apply CSDPLAN when building a storage server that runs erasure coding. The storage architect inputs into CSDPLAN-EC the computing power of the host CPU and the CSD. CSDPLAN-EC provides a BEP using a specialized analysis model based on CSDPLAN when executing erasure coding, which the storage architect can use as a guideline.

Fig. 1. An overview of CSDPLAN-EC.

2. Extended of CSDPLAN for EC Offloading

We explain the CSDPLAN-EC, which is the methodology for using CSDPLAN when performing erasure coding. Fig. 2(a) and (b) depict the execution time phase when performing erasure coding for M data blocks in the SSD system and CSD system equipped with M number of devices, respectively. In each figure, erasure coding time (T$_{\mathrm{SSD-ec}}$, T$_{\mathrm{CSD-ec}}$) is (n, k) RS code execution time once. In other words, it means that erasure coding is executed M times in each system equipped with M number of devices. In Addition, it is assumed that block data transfer is possible asynchronously with erasure coding.

Therefore, in order to apply CSDPLAN modeling, as shown in Fig. 2(a), the execution time when erasure coding is performed for M blocks in an SSD system with M SSDs is as follows.

On the other hand, in the CSD system equipped with M number of CSDs, as shown in Fig. 2(b), parallel processing is performed in each CSD, as follows.

To find the BEP, we derive the mathematical model as follows:

By assumption, data transfer time is only required once for the first time, so it is ignored for simplicity of the model. Then,

Therefore, BEP is affected by the computational power of the host CPU and CSD, and BEP changes according to this are as follows.

Fig. 2. Comparison of erasure coding execution time between SSD system and CSD system.

1. Evaluation of Commercial CSDs

We extensively evaluated the computing and I/O performance of two notable commercial CSDs, namely SmartSSD and Newport CSD. The evaluation was conducted on a host server that had dual AMD EPYC$^{\mathrm{TM}}$ 7352 CPUs, 256GB DRAM and was running CentOS 7.9. The comprehensive specifications of the host server and the CSDs can be found in Tables1 and 2 .

We employed various methodologies to evaluate the performance of CSDs, depending on their unique hardware designs. To evaluate computing performance, we developed in-house big data analysis kernels\footnote{$^{}$The data size in parentheses is the workload size.} such as Count (4.8 GB), Vector Addition (4.8 GB), Array Merge (4.8 GB), and Page Rank (0.2 GB). In contrast, we used the FIO benchmark ^[31] for the host and Newport CSD to measure external and internal I/O bandwidth. For SmartSSD, we used a bandwidth measurement kernel program ^[32] To set up the FIO benchmark, we chose LibAIO\footnote{$^{}$Linux-native Asynchronous I/O Access Library} and selected the direct option while configuring it for a 1MB request size, 64 queue depth, and sequential pattern.

1) I/O Bandwidth Capability: In Fig. 4, we compare the internal and external I/O bandwidth of the CSDs. The internal I/O bandwidth pertains to the rate at which the device’s kernel accesses data in the NAND flash, whereas external I/O bandwidth refers to the speed at which the host’s kernel accesses data in the NAND flash. For SmartSSD, we observed that the external bandwidth for reads is roughly 1.18$\times $ higher than the internal bandwidth. On the other hand, for writes, the internal bandwidth is about 1.36$\times $ higher than the external bandwidth. We attribute these constraints to the PCIe bus connecting the SSD and the FPGA, which limits the maximum internal bandwidth.

In contrast, when testing the Newport CSD, we made an unexpected observation: the external bandwidth for both read and write workloads was remarkably higher than the internal bandwidth by a factor of 2.28 and 1.33, respectively. This finding suggests that the internal bandwidth of commercial CSDs may be lower than their external I/O bandwidth. Therefore, kernels executed within the device cannot always rely on having higher I/O bandwidth.

2) Computational Capability: We evaluated the computing power of an SSD system (1) using a single core of the CPU in the host server and two CSD systems using SmartSSD and Newport CSD’s processing engine when running the analysis kernel. Fig. 3 shows the execution times of the CSD systems for analysis kernel workloads normalized to the SSD system (1). For the convenience of description\footnote{$^{}$ Hereinafter, we use SSD system, CSD system, SSD(n), CSD-S, and CSD-N interchangeably.}, the CSD system equipped with SmartSSD and Newport CSD and the SSD system (1) are expressed as CSD-S, CSD-N, and SSD(1), respectively. We conducted this experiment with the assumption that all data required for kernel execution was already loaded entirely in DRAM, thus eliminating any I/O time involvement. To maximize the performance, CSD-S used all FPGA optimization techniques ^[33-36], such as local memory buffer, loop pipelining, and multiple compute units, while CSD-N adopted multithreading.

Overall, both CSD-S and CSD-N are slower than the SSD (1) for compute-intensive workloads. In all analysis kernels, CSD-S was 1.6$\times $, 4.6$\times $, 28.1$\times $, and 15.4$\times $ slower than SSD(1), respectively, and CSD-N was 1.7$\times $, 6.3$\times $, 2.8$\times $, and 1.3$\times $ slower than SSD(1), respectively. Therefore, the representative commercial CSDs have low computing power, and naively offloading computation to the device might not be as effective as expected.

Fig. 3. Comparison of the computational power of SSD system and CSD system.

Fig. 4. Bandwidth evaluation of SmartSSD and Newport CSD.

2. Evaluation of CSDPLAN for Erasure Coding

We evaluated CSDPLAN when erasure coding offloaded to CSD through the change in break-even point (BEP) according to the size and number of blocks and the computational power of CSD. The block size means the size of the data block to execute RS code, and the number of blocks is the number of blocks created after the completion of RS code execution. When (n, k) RS code is executed, n + k blocks are created.

We evaluated erasure coding on both the SSD system and the CSD system with a single device. When running erasure coding, the SSD system uses the host CPU single-threaded, whereas the CSD system uses multi-threading. In addition, to use the same erasure coding library ^[37] in the Linux environment for both the SSD system and CSD system, the CSD system used Newport CSD running operating system.

1) Impact of Block Size: Fig. 5(a) shows the execution time of the SSD system and CSD system according to the block size when encoding 24 blocks with (16, 8) RS code. As the block size increases, SSD(1) has little change from 3 to 4 seconds, while CSD-N slows down significantly from 4 seconds to 10 seconds. This is because CSD’s computational power is weaker than that of the host CPU, so it is more affected by the block size.

Fig. 5(b) shows the BEP related to Fig. 5(a). When the block size is 16 MB, the BEP is 2, and when the block size is 128 MB, the BEP increases to 3. Compared to the block size increase, the BEP increase is insignificant.

2) Impact of Number of Blocks: Fig. 6(a) shows the execution time for encoding 64 MB data blocks from 12 to 48 blocks. In (n, k) RS code, n = 2k, i.e., (8, 4) RS code when encoding with 12 blocks. When the number of blocks increases from 12 to 48, SSD(1) shows little change from 3 seconds to 4.8 seconds, while CSD-N increases significantly from 3.8 seconds to 22.3 seconds. Fig. 6(b) shows the BEP change of CSD-N related to Fig. 7. BEP more than doubles from 2 to 5. In erasure coding, it can be seen that BEP is more affected by the number of blocks than the size of blocks.

3) Impact of Computational Power: Fig. 7 shows the BEP as a function of computational power when executing the (32, 16) RS code with a 64 MB data block. Fig. 7(a) shows the BEP as the computational power of the CSD increases. It can be seen that the BEP decreases to 1 when the computational power is increased by a factor of 5. Through this, it is possible to know the required degree of computational power improvement of CSD to find an appropriate BEP. Fig. 7(b) shows the BEP with increasing computational power of the CSD and decreasing computational power of the host CPU. In Fig. 7(a), the BEP becomes 1 when the CSD computational power is increased by 5$\times $, but the BEP becomes 1 when the CSD computational power is increased by 2.3$\times $, and the host CPU computational power is decreased by 2.3$\times $. This shows that the BEP can be found by combining the host CPU and CSD.

Fig. 5. Changes of erasure coding execution time and break-even point according to block size.

Fig. 6. Changes of erasure coding execution time and break-even point according to the number of blocks.

Fig. 7. Change in break-even point with computational power: (a) Increase in computational power of CSD; (b) Increase in computational power of CSD and decrease in computational power of host CPU.