Browse Prior Art Database

Methods and Apparatus on shedding and delaying data in real time data processing systems

IP.com Disclosure Number: IPCOM000244607D
Publication Date: 2015-Dec-28

Publishing Venue

The IP.com Prior Art Database

Abstract

In this paper, we consider the problem of allocating CPU-cores in a Spark Streaming infrastructure in the context of two types of queries, namely primary and optional, that are associated with high- and low-priority analysis, respectively. We develop a controller, iBLOC, that adjusts the block sizes of streaming jobs on the fly and parallelism level of jobs, according to the input data rates and the query priorities. Our evaluation shows that we can achieve significant CPU-core savings from the primary query type such that multiple queries can run together without impairing their latency constraints, in comparison to a static block-sizing scheme.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 97% of the total text.

Page 01 of 10

Methods and Apparatus on shedding and delaying data in real time data processing systems

!" # $ " %

&" '

("

) *+,

#

- "

.

#
*+, # #

# / # 0

*+,
* 1 $ 1 #

*+, / $ $

$ 2 1 $ 3 $

*+, * #

$ $ 2 ) 2 $

0 $ *+,

# 2 2

*+, $ 0 +

$ 0 # 4 5" % # $

# # -"

) # 6

# $ 0 2

# # 0 7

89%* 2 $ 0 $

$ )

$ : $ 2 *+,

89%* #

7 89%* 1 $ 1 %

89%* 2 $ 0

#

$ 0

%


Page 02 of 10

# *+,

89%* # #

# 1
*+, 0 $ 0 $

$ 1 )) 8

$ 0 89%*

1 )) 1 ))) 1 );

2

1 1 $ 1 6 $ 2 *+, 0

1 ; $ 1 ;)

)) 1

) 3* ,'3

) 1 $ 1 -"

1 $ 1 1 $ $ 5"

) $

$ 2

2 *+,

$ 0

= 8 $ 8 ? 1 $ 1

1 $ 1 6

$

$ 2

# 1 $

@ A(( : B 3

# $

# C $ = $

$ # $ $ $ #

D $ D:C = # 2 E

= 2 2 E F G !H =

2 2 # 2 1 $

>

2

) 1 $ # 2 $ D:C $

3 $ *+,

$

# *+, $ 3 $

$ 2 )

2 $

$ $ =

$ 5" 2 0

$

!


Page 03 of 10

2 2

2

$ 2 $ C D

2 E > C E $ D

8 1 / 6 . 3

7 $

=

$ 89%* # 1 $

A(( : 7

$ C F (( -(( = $

D F 1 $1 # !

5 J8 #

1 ); =

! 0 *+, 2 %

$ @C F (( ! @ B @ BB

$ $ : $ 2 = 2

*+, $ >

*+, #

% $ @C F -(( ! @ B @ BB

$ *+, $

*+, # 7

$ $

2 *+, =
*+, 2
*+,

# $

$

$ *+, 2 %

$

2 2 )

89%*

A


Page 04 of 10

))) )89%*

) 89%*

2 $ 2 $ )

89%* $ 2 2

   89%* 6 * . 3 9
* A 1 $ 1 ) 89%*

= *

$ C 2

$ 2

=

2 )

2 $ 2

: $ 2 =

K

$ K 7

$ = $

$ $

# = 2

$ $

F L C L $

89%* $

@ B

3# 89%* $ C 1 ˆ $ˆ

6

ˆ $ˆ . 3

1 ))) 8

0 $

F

D $ 2 =

$

.

C

                   N = CN F O L D ( P O ∗ $

$ O F (- C

$

5


Page 05 of 10

* >

# R

$

89%* 6

8 . 3

89%* $

$

6
ˆF @ S TB L ˆSU T L S $

ˆ

T

T

A 89%*

* 9 *

) $ C

89%* 2

= 89%*

1 $ 1 2

:

> C

$ : $

: $ * # 89%*

C

(4LCN % 2 % 89%* 2 7

2

2 $ 0 )

); 3;=9,= )%R

) 1 $ 1 ,

# 2

89%* .

2 *+,

2 89%* $ @B2 @ B

2

ˆ

F @ S TB L $

S

U T L $ S

Q


Page 06 of 10

=

  7 1 $ A ( @;. B , 5 (5 ;. ;. # 5 J8 = 0 3*! 0 % ;. 1 $ 1 $ $ 1 $ 1 $ $

=

)

2

7 * $ V" 7 J

$ +

A ((( : A(∗

1 $

8 ? %

7 1 )) 8

2 > 89%*

7 A!Q

: !Q : !(

*+, 2

7 1 $ # -

1 $ $

5 2 *+,

89%* $

7 D F 2 #

F D =

$ @ 5 @ BB

$

V


Page 07 of 10

# $ *+, 9 $ *+,

@ 5 @ BB

@B O (- #

$ @ B7

*+, %

$ $ # $

- 1 $ $

7

2

@ 5 @ BB %

)


* ?

>

. 2

, 2

> !Q : &(( :

Q( $ $ $

*+, 2
= *+,

*+, @ Q @ BB = 2 $ @ Q @ BB

2 1

# 2 .

1 $ #

2

&


Page 08 of 10

/ + 2

$ 6 @ B

2

K @ B 0

*+, $ *+,

2 6 2

2 2

2

2

2 7

# !Q : &(( :

Q( 2

2 *+, 2

0 V $ 2

1 V @ B

2 2

*+, 9 $

V @ B *+, ) 2 2 $

2

V @ B

; '39= 3/ 7%'W

' 1 $ 1 - Q" 133+ !" 1

R ("

= $

= 0

# 2

$ 5"

$ $ 2 )

0 $ 2

   0 2 $

V"

-


Page 09 of 10

2 A" ) &" $ #

2 = $

!" . $

A " 7 # $

# # >

0 #

;) *%R*9,1)%R =R/ /)1*,11)%R

) #

*+,

7 89%* 2 $ 0

$ # *+, 1 89%*

$ #

89%* $ # %

89%* #

$ 0 = $

89%* 0 # # % $ 1 $ 1 ) $

$
'3 3'3R*31

" = + @= +B # $ !((-

!"' * 0 . . 3 W $ + + 0 ) ) =*.
1)J.%/
&!QX&AV !( A

A" = / ? J $ . ' = 2 )
=*. 1)J.%/ 5(XQ !((A

5" / < Y ) 1 1 1 $ =

0 ) =*. 1 ** V6 X V6 A !(

" 8 > . < Y J ' * 8 1 7 9 9 Y * 6

) =*. 1 ** VAX&5 !( (

!" 1 > ? > ? / Z 8 > $ 6


* 0 ) )*/3 7 $ 5 XQ !( (

A" 3 W $ * . + + 0 % .

/ 1 + 1 9 J ) $ * 7 $ !( !

5" 3 W $ 7 7 [ > ; / W + + 0 1[+'6

# ) )*/3 -5(X-Q !(

Q" W W = 8 . W %0 ; 3 * Z ? ? $

/ 0 # ) 1)J.%/ 45AX4Q5 !( 5

4


Page 10 of 10

V" / J . .1 ' ) . ) + 8 . = R 6 =
) =*. 1%1+ 5A4X5QQ !( A

&" + ' + 0 ? 9 ? 1 . ' . 7 . ) 1 0

R $ ) )*/3 54 !((V

-" 3 ?...