ZFS Benchmark

#Details

date

Sa 16. Jan 14:30:47 CET 2016

hostname

dl380

uname -a

Linux dl380 3.16.0-57-generic #77~14.04.1-Ubuntu SMP Thu Dec 17 23:20:00 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

free -h

             Gesamt Belegt Frei Gemeinsam Puffer Cache
Speicher:       7,8G       629M       7,2G       896K        23M       117M
-/+ Puffer/Cache:       488M       7,3G
Auslagerungsdatei:       279M         0B       279M

fdisk -l 2>&1 | grep "^Disk"

Disk /dev/sda: 1000.2 GB, 1000171331584 bytes
Disk /dev/sdb: 1000.2 GB, 1000171331584 bytes
Disk /dev/sdc: 256.0 GB, 256022937600 bytes
Disk /dev/sdd: 250.0 GB, 250023444480 bytes
Disk /dev/sde: 1000.2 GB, 1000171331584 bytes
Disk /dev/sdf: 1000.2 GB, 1000171331584 bytes
Disk /dev/sdg: 256.0 GB, 256022937600 bytes
Disk /dev/sdh: 250.0 GB, 250023444480 bytes

...more

Test Size
16 GB

#Prepare

/bin/echo "" > bench/boonie.csv

#Disks

Disk	Benchmark
/dev/sda	/dev/sda: Timing O_DIRECT cached reads: 622 MB in 2.00 seconds = 310.68 MB/sec Timing O_DIRECT disk reads: 326 MB in 3.02 seconds = 108.00 MB/sec
/dev/sdb	/dev/sdb: Timing O_DIRECT cached reads: 1334 MB in 2.00 seconds = 666.14 MB/sec Timing O_DIRECT disk reads: 312 MB in 3.01 seconds = 103.75 MB/sec
/dev/sdc	/dev/sdc: Timing O_DIRECT cached reads: 474 MB in 2.01 seconds = 236.40 MB/sec Timing O_DIRECT disk reads: 724 MB in 3.01 seconds = 240.86 MB/sec
/dev/sdd	/dev/sdd: Timing O_DIRECT cached reads: 474 MB in 2.00 seconds = 236.50 MB/sec Timing O_DIRECT disk reads: 730 MB in 3.00 seconds = 243.05 MB/sec
/dev/sde	/dev/sde: Timing O_DIRECT cached reads: 474 MB in 2.01 seconds = 236.31 MB/sec Timing O_DIRECT disk reads: 318 MB in 3.01 seconds = 105.51 MB/sec
/dev/sdf	/dev/sdf: Timing O_DIRECT cached reads: 878 MB in 2.00 seconds = 438.52 MB/sec Timing O_DIRECT disk reads: 322 MB in 3.02 seconds = 106.77 MB/sec
/dev/sdg	/dev/sdg: Timing O_DIRECT cached reads: 474 MB in 2.00 seconds = 236.68 MB/sec Timing O_DIRECT disk reads: 722 MB in 3.01 seconds = 240.21 MB/sec
/dev/sdh	/dev/sdh: Timing O_DIRECT cached reads: 474 MB in 2.01 seconds = 236.29 MB/sec Timing O_DIRECT disk reads: 730 MB in 3.00 seconds = 242.99 MB/sec

#Benchmarks

1. Benchmark:

/sbin/zpool create tank /dev/sda

2. Benchmark:

/sbin/zpool create tank /dev/sda /dev/sdb

3. Benchmark:

/sbin/zpool create tank /dev/sda /dev/sdb /dev/sde

4. Benchmark:

/sbin/zpool create tank /dev/sda /dev/sdb /dev/sde /dev/sdf

5. Benchmark:

/sbin/zpool create tank /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

6. Benchmark:

/sbin/zpool create tank mirror /dev/sda /dev/sdb

7. Benchmark:

/sbin/zpool create tank mirror /dev/sda /dev/sde

8. Benchmark:

/sbin/zpool create tank mirror /dev/sda /dev/sdb mirror /dev/sde /dev/sdf

9. Benchmark:

/sbin/zpool create tank mirror /dev/sda /dev/sdb mirror /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc

10. Benchmark:

/sbin/zpool create tank mirror /dev/sda /dev/sdb mirror /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc /dev/sdg

11. Benchmark:

/sbin/zpool create tank mirror /dev/sda /dev/sdb mirror /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc /dev/sdg
/sbin/zpool add -f tank log /dev/sdd

12. Benchmark:

/sbin/zpool create tank raidz /dev/sda /dev/sdb /dev/sde /dev/sdf

13. Benchmark:

/sbin/zpool create tank raidz /dev/sda /dev/sdb /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc

14. Benchmark:

/sbin/zpool create tank raidz /dev/sda /dev/sdb /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc /dev/sdg

15. Benchmark:

/sbin/zpool create tank raidz /dev/sda /dev/sdb /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc /dev/sdg
/sbin/zpool add -f tank log /dev/sdd

16. Benchmark:

/sbin/zpool create tank raidz2 /dev/sda /dev/sdb /dev/sde /dev/sdf

17. Benchmark:

/sbin/zpool create tank raidz2 /dev/sda /dev/sdb /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc

18. Benchmark:

/sbin/zpool create tank raidz2 /dev/sda /dev/sdb /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc /dev/sdg

19. Benchmark:

/sbin/zpool create tank raidz2 /dev/sda /dev/sdb /dev/sde /dev/sdf
/sbin/zpool add tank cache /dev/sdc /dev/sdg
/sbin/zpool add -f tank log /dev/sdd

#Benchmark Matrix

Benchmark	Pool
1.1	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 sda ONLINE 0 0 0 errors: No known data errors
1.2	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 errors: No known data errors
1.3	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 errors: No known data errors
1.4	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 errors: No known data errors
1.5	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors
2.1	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 errors: No known data errors
2.2	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sde ONLINE 0 0 0 errors: No known data errors
2.3	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 errors: No known data errors
2.4	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 cache sdc ONLINE 0 0 0 errors: No known data errors
2.5	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 cache sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors
2.6	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 logs sdd ONLINE 0 0 0 cache sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors
3.1	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 errors: No known data errors
3.2	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 cache sdc ONLINE 0 0 0 errors: No known data errors
3.3	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 cache sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors
3.4	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 logs sdd ONLINE 0 0 0 cache sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors
4.1	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 errors: No known data errors
4.2	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 cache sdc ONLINE 0 0 0 errors: No known data errors
4.3	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 cache sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors
4.4	pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 logs sdd ONLINE 0 0 0 cache sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 errors: No known data errors

/usr/bin/bon_csv2html bench/boonie.csv > bench/boonie.html

Version 1.97		Sequential Output						Sequential Input				Random Seeks			Sequential Create						Random Create
	Size	Per Char		Block		Rewrite		Per Char		Block				Num Files	Create		Read		Delete		Create		Read		Delete
		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU		/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU
dl380_1.1	32G	141	99	97849	28	44364	8	402	91	93455	6	93.4	7	128	16568	99	+++++	+++	16236	99	19574	99	+++++	+++	16229	99
	Latency	61300us		57382us		3737ms		342ms		1021ms		724ms		Latency	156ms		480us		12479us		98335us		196us		10444us
dl380_1.2	32G	141	99	138336	36	72670	14	398	91	164764	11	177.1	13	128	16090	99	+++++	+++	17516	99	20145	99	+++++	+++	17481	99
	Latency	62121us		60778us		2136ms		691ms		623ms		188ms		Latency	171ms		431us		2128us		94460us		65us		1891us
dl380_1.3	32G	140	99	183301	43	85265	16	427	98	227206	15	240.5	18	128	16374	99	+++++	+++	16698	99	19124	99	+++++	+++	16953	99
	Latency	62304us		54434us		2250ms		142ms		777ms		190ms		Latency	169ms		313us		578us		96169us		20us		1401us
dl380_1.4	32G	140	99	237496	52	115436	23	409	93	282633	19	283.3	21	128	16421	99	+++++	+++	16841	99	18321	99	+++++	+++	17389	99
	Latency	62806us		22012us		2124ms		343ms		437ms		194ms		Latency	165ms		608us		2526us		92224us		153us		1635us
dl380_1.5	32G	139	99	381242	65	160050	32	433	99	411817	28	368.1	26	128	16119	99	+++++	+++	15874	99	18450	99	+++++	+++	15432	99
	Latency	65179us		24726us		2033ms		64256us		536ms		192ms		Latency	167ms		627us		1173us		94552us		199us		3726us
dl380_2.1	32G	142	99	54925	21	39888	7	412	94	124546	9	184.6	14	128	16694	99	+++++	+++	16877	99	19114	99	+++++	+++	16419	99
	Latency	64070us		41582us		2785ms		185ms		1297ms		660ms		Latency	165ms		476us		934us		77781us		56us		689us
dl380_2.2	32G	141	99	75985	25	43506	8	387	88	116400	8	152.6	11	128	17155	99	+++++	+++	16310	99	19890	99	+++++	+++	14896	99
	Latency	67152us		61543us		4271ms		711ms		1001ms		479ms		Latency	147ms		309us		742us		106ms		184us		687us
dl380_2.3	32G	141	99	127225	35	72400	14	418	96	222389	15	263.1	19	128	16339	99	+++++	+++	16445	99	17850	99	+++++	+++	15215	99
	Latency	65837us		56941us		2008ms		131ms		333ms		141ms		Latency	168ms		474us		3140us		96852us		178us		776us
dl380_2.4	32G	141	99	126219	34	77749	15	420	98	222516	16	755.2	40	128	16459	99	+++++	+++	17968	99	17964	99	+++++	+++	16458	99
	Latency	64082us		35241us		1598ms		60011us		754ms		215ms		Latency	167ms		460us		764us		98468us		196us		7357us
dl380_2.4_c	32G	135	99	129283	35	77823	16	424	97	231768	16	756.9	41	128	16366	99	+++++	+++	14802	99	16211	99	+++++	+++	13510	99
	Latency	67748us		32324us		2103ms		115ms		568ms		157ms		Latency	161ms		330us		16185us		71208us		187us		3716us
dl380_2.5	32G	139	99	123070	34	71489	14	420	97	200350	14	699.5	38	128	17253	99	+++++	+++	17673	99	16531	99	+++++	+++	14227	99
	Latency	63937us		31930us		2094ms		92368us		702ms		355ms		Latency	150ms		351us		1189us		80427us		181us		1358us
dl380_2.5_c	32G	133	99	123501	34	76880	15	428	99	225573	16	677.2	38	128	16767	99	+++++	+++	17855	99	19269	99	+++++	+++	14631	99
	Latency	63400us		47715us		1977ms		62947us		794ms		189ms		Latency	139ms		529us		1917us		124ms		170us		2629us
dl380_2.6	32G	138	99	135117	36	71580	14	423	98	190964	13	753.7	41	128	16104	99	+++++	+++	17847	99	20543	99	+++++	+++	15199	99
	Latency	60867us		78998us		1425ms		95215us		1021ms		307ms		Latency	171ms		319us		6846us		92732us		54us		10552us
dl380_2.6_c	32G	132	99	121844	34	69558	14	404	94	221041	16	699.0	38	128	16476	99	+++++	+++	18882	99	19998	99	+++++	+++	15101	99
	Latency	82988us		31221us		1752ms		326ms		3440ms		375ms		Latency	157ms		317us		511us		88934us		172us		3126us
dl380_3.1	32G	140	99	196248	47	94393	19	411	95	238990	16	144.7	11	128	17694	99	+++++	+++	17287	99	19714	99	+++++	+++	16174	99
	Latency	84813us		24621us		4147ms		314ms		694ms		186ms		Latency	134ms		319us		5429us		106ms		164us		791us
dl380_3.2	32G	142	99	194129	46	92417	19	396	91	237380	18	338.2	22	128	15924	99	+++++	+++	17274	99	19089	99	+++++	+++	14204	99
	Latency	62326us		20796us		2165ms		648ms		793ms		227ms		Latency	175ms		467us		571us		113ms		140us		1484us
dl380_3.2_c	32G	132	99	199150	47	90332	19	401	93	239989	17	275.4	19	128	16581	99	+++++	+++	16564	99	16039	99	+++++	+++	16591	99
	Latency	88019us		37428us		2471ms		522ms		785ms		238ms		Latency	156ms		313us		737us		70065us		145us		4364us
dl380_3.3	32G	138	99	206129	48	91522	19	376	87	231863	17	286.7	20	128	19572	99	+++++	+++	18445	99	19581	99	+++++	+++	14809	99
	Latency	64083us		35517us		2007ms		959ms		949ms		240ms		Latency	102ms		411us		952us		122ms		218us		3372us
dl380_3.3_c	32G	131	99	193545	45	90501	19	397	92	243199	18	248.2	18	128	16532	99	+++++	+++	16151	99	18343	99	+++++	+++	18565	99
	Latency	68865us		18034us		2096ms		575ms		1246ms		207ms		Latency	160ms		312us		3870us		103ms		65us		790us
dl380_3.4	32G	138	99	201960	47	95133	20	416	96	238301	18	344.7	23	128	16214	99	+++++	+++	17313	99	15686	99	+++++	+++	18413	99
	Latency	66618us		29320us		999ms		238ms		536ms		197ms		Latency	172ms		499us		1421us		76159us		20us		3612us
dl380_3.4_c	32G	130	99	205910	48	94888	20	411	94	250719	18	277.8	19	128	16222	99	+++++	+++	14135	99	16040	99	+++++	+++	16767	99
	Latency	64172us		28439us		1433ms		326ms		1007ms		200ms		Latency	162ms		565us		1021us		70149us		80us		1756us
dl380_4.1	32G	136	99	133965	37	63985	13	409	96	174624	12	144.4	11	128	16849	99	+++++	+++	16401	99	17947	99	+++++	+++	18964	99
	Latency	66355us		37974us		2003ms		247ms		769ms		259ms		Latency	149ms		454us		1163us		91621us		175us		791us
dl380_4.2	32G	139	99	134505	37	62145	13	422	98	171561	12	345.5	24	128	16362	99	+++++	+++	16975	99	17744	99	+++++	+++	14716	99
	Latency	62688us		31405us		1473ms		69941us		999ms		268ms		Latency	164ms		472us		1427us		94648us		134us		804us
dl380_4.2_c	32G	131	99	134627	38	60457	13	379	89	172934	12	328.4	23	128	15968	99	+++++	+++	16259	99	15970	99	+++++	+++	15436	99
	Latency	66384us		25818us		2028ms		799ms		589ms		223ms		Latency	171ms		458us		823us		68709us		173us		2364us
dl380_4.3	32G	140	99	129538	37	61224	13	390	91	158747	11	349.8	24	128	16010	99	+++++	+++	15894	99	17960	99	+++++	+++	15590	99
	Latency	63009us		29532us		2018ms		421ms		606ms		225ms		Latency	175ms		512us		789us		97450us		241us		1699us
dl380_4.3_c	32G	130	99	134708	37	65048	14	406	93	185901	13	384.8	25	128	16086	99	+++++	+++	17676	99	20173	99	+++++	+++	13483	99
	Latency	71897us		30215us		1752ms		245ms		331ms		226ms		Latency	168ms		545us		3934us		101ms		182us		2754us
dl380_4.4	32G	138	99	133463	37	64845	14	407	94	171688	13	366.5	25	128	16403	99	+++++	+++	16453	99	19505	99	+++++	+++	14417	99
	Latency	60389us		21026us		1627ms		382ms		517ms		223ms		Latency	166ms		720us		6047us		122ms		177us		4998us
dl380_4.4_c	32G	131	99	129288	36	63658	13	412	95	187251	13	375.8	25	128	16115	99	+++++	+++	17345	99	17184	99	+++++	+++	14396	99
	Latency	84861us		17395us		1374ms		350ms		869ms		304ms		Latency	163ms		421us		777us		88193us		195us		1024us